Hello,
NTA 4.0 RC is available for all customers that are covered under maintenance.
This is a whopper of a release for NetFlow Traffic Analyzer. NTA 4.0 brings one minute granularity for all data! NTA no longer aggregates data over time. This is a big change for us and a big change for you! With this change comes some critical information that you need to take into account.
First, we are no longer storing flow data in SQL, we have developed our own Flow Storage Database that will now be responsible for the processing and storage of flow data. Secondly we have introduced web based reporting for NTA.
Pre-requisites for NTA 4.0:
- 64bit Environment is required, if you have more than one poller, make sure they are all on 64bit OS’s.
- You will be able to install a 32bit version of NTA however it will not use the new Flow Storage Database
- NPM 10.6 is required, NPM 10.6 has the new web based reporting engine which is required for NTA 4.0
- NTA Flow Storage Database install is required and should be done before upgrading / installing NTA on the pollers.
- Connection to Orion database is required, CBQoS data and some additional low level information is still stored in this DB.
Hardware requirements for NTA 4.0
- 32Bit: (does not provide the new storage and the scalability benefits that are available on 64bits)
Same as for NTA 3.11 (it means we rely on NPM’s requirements for all components)
- 64Bit: (required to get the escalability benefits of NTA 4.0)
Poller: Same as for Orion NPM, see the Admin Guide p16.
Orion Database: Same as for Orion NPM, see the Admin Guide p16.
Flow storage Database:
- Quad core 3ghz Processor or better
- 16 Gigs of Ram and should be increased as database size increases to ensure optimal performance
- 20GB (minimum) of storage on 7200rpm disk
- About 100GB for each sustained 1000 flows/s with 30 days retention period’
- 64 bit Windows Server 2003 SP4 or newer.
- Itanium-based servers are not supported
"Where do I install the Flow Storage Database?
The Flow Storage Database can be installed on a poller (Main or additional), it can also be installed on a remote machine or VM with in your environment. The Flow Storage Server must be installed first before upgrading or installing NTA 4.0 on your pollers. Given the overhead on CPU and Memory that capturing and processing flow data requires we are strongly recommending that you install the Flow Storage DB on a separate VM or physical machine with in your network. Local poller installations are supported but will not see same type of performance increases that are capable by running the Flow Storage Database on a dedicated machine.
FAQ about the Flow Storage Database:
Where should I install my Flow Storage Database?
- We strongly recommend a dedicated machine for the NTA Flow Storage Database, Orion/NPM performance will be improved and NTA performance will be significantly improved
Can I install the Flow Storage Database on a local poller (Main or additional)
- Yes you can install this DB on a local poller, when doing so we recommend installation on a separate partition/dedicated disk. We also recommend that this machine be on the higher end of specs. 8 cores @3GHz, 16-32gb of ram and a fast disk.
Can I install the Flow Storage Database on the Orion/NPM server that is hosting the SQL DB?
- Yes you can install the FSDB on this box, this is not recommended. If your only option is to run on this machine please ensure that the NTA FSDB is installed on a separate partition with a dedicated disk for storage. Highest end machine required for this deployment 16+ cores @3GHz, 64-128GB of ram. If you have a low number of PDU’s per second (<1k) you will not need to have a high end machine
Do I need SQL installed on the machine that the Flow Storage Database will be installed on?
- No, NTA's Flow Storage Database is not a SQL DB.
What does my physical or VM server that I will install the NTA FSDB on need to have installed?
- A 64bit Windows Server 2003 SP4 or newer.
- .NET framework 3.5sp1 and 4.0 (Included in the installer if you have not previously installed them)
Do I need to open specific ports on the Flow Storage Database server?
- Yes, it is necessary to open tcp 1433 between the new data storage and the existing SQL server, as well as tcp 17777 between each of the pollers to the new storage server.
Do I need a connection to the Orion/NPM database?
- Yes, if you are installing Flow Storage DB to dedicated machine, during the configuration wizard you will be required to enter the hostname/IP and credentials of the Orion/NPM Database, NTA requires this connection for CBQoS and other information provided by NPM
How do I determine the right amount of CPU/MEM for my Flow Storage Database?
- Type following line into command line on all of your pollers (during business hours to capture the higher load):
typeperf "\SolarWinds NetFlow Averages\PDUs Processed per second" -sc 1
If the total count from all pollers is less than 5k, we recommend that you have a minimum of 4 cores @3ghz for processing and 8-16GB of Ram. If that number is bigger than 5k we recommend 8 cores @3gzh for processing and 16-32GB of Ram
Note that the value returned by this command is the average flow/sec for this poller, over the last 5 minutes
How do I determine the right amount of Storage for my NTA 4.0 Flow Storage Database, based on the amount of flows per sec?
- Reference the Average PDU’s processed per second number (see above). If you are near 2k we recommend a minimum of 200GB of local NTFS storage for a 30 day (default) retention period. Please also take into account how much data you will be migrating from your previous SQL flow storage, you will be informed on how much space is required for the migration (if you choose to) during the configuration wizard. You can also gauge this number by running this query on your Orion SQL DB (Insert Query Here). If your average PDU’s per second processed per second are between 4k and 5k we are recommending a minimum of 0.5-1.5 Terabyte of local NTFS storage. If you are near 10k we will recommend 2-4 terabytes of local NTFS storage. If you are running a huge environment that is capturing well above 10k as in 20-45k per second you will need 4-8 terabytes of storage
How do I determine the right amount of Storage for my NTA 4.0 Flow Storage Database, based on the my NTA 3.x usage?
- This procedures applies to customers activelly using NTA 3.x
- Run the below script. The result should look like this:
- This is the *estimated* size (in GB) required by the NTA 4.0 FSDB, after upgrade from 3.x and for the currently configured retention period (default=30 days), assuming the flow pattern remains similar to what it is in NTA 3.x.
- Note: This size is calculated based on lasthour activity, so there can be significant variations, as the last hour may not reflect longer periods of time. We recommend customers to use this as a general guideline and perform their own testing in their own environment.
DECLARE@FSDB_row_size INT;DECLARE@detailed_row_count BIGINT;DECLARE@retention_period BIGINT;DECLARE@AmountOfDetailTablesPerNode ASINTSET@FSDB_row_size = 126;----------------------------------------------------- ** Select settings from NTA Global settings **SELECT@AmountOfDetailTablesPerNode = CONVERT(INT, Value)FROMNetFlowGlobalSettingsWHEREKeyName ='RetainUncompressedDataIn15MinuteIncrements'SELECT@retention_period = value*24*60*60 FROMdbo.NetFlowGlobalSettings WHEREKeyName='RetainCompressedDataInDays'-- ** Get all detail tables **SELECT* INTO#AllDetailTables FROM( SELECTSUBSTRING(name,15,PATINDEX('%[0-9][_]%', name)-14) ASNodeID, RIGHT(name, LEN(name) - PATINDEX('%[0-9][_]%', name)-1) asInterval, name, id FROMdbo.sysobjects WHEREnameLIKE'NetFlowDetail[_][0-9]%'ANDOBJECTPROPERTY(id, N'IsUserTable') = 1) ASA-- ** Get only first [amount of 15 minutes intervals] detail tables per each node **DECLARE@script asNVARCHAR(max)SET@script ='SELECT * INTO ##DetailTables FROM #AllDetailTables aWHERE name IN( SELECT TOP '+ CONVERT(nvarchar, @AmountOfDetailTablesPerNode) + ' name FROM #AllDetailTables b WHERE b.NodeID=a.NodeID ORDER BY Interval)ORDER BY name'EXECUTEsp_executesql @script-- ** Get sum of all rows in given detail tables **SELECT@detailed_row_count = SUM(row_count)FROMsys.dm_db_partition_stats stats JOIN##DetailTables dt ON(stats.object_id = dt.id)-- ** Calculate target size based on amount of rows in detail tables **SELECT((@detailed_row_count / (@AmountOfDetailTablesPerNode*15*60)) * @FSDB_row_size*@retention_period) / 1024 / 1024 / 1024 AS[Required_FSDB_Disk_Size_GB];DROPTABLE#AllDetailTablesDROPTABLE##DetailTables
Why is so much storage required?
- With one minute granularity for the entirety of the retention period the storage requirements for NTA have increased 8-10x for the same retention period when compared to previous SQL storage of flow data. However, data from SQL server will be migrated so you will get some extra free space on your SQL server.
Can I change the default retention period?
- Yes, NTA 4.0 will default the retention period to 30 days, you can change that number at any time on NTA settings page, and we have found that most customers only need 2 weeks of historical data. There are also some customers who are interested in 6months to a year for their data retention. In the future we will be providing more options to our users on how to manage the retention period as well as the database growth.
What can I do if my Flow Storage has reached capacity?
- NTA will alert you if your Flow Storage Database nears or reaches capacity, in this case we recommend that you adjust your retention period to a shorter period. If you reached capacity on day 27 of your retention period, adjust your retention period accordingly and the data will be trimmed from your DB automatically the next time the midnight maintenance runs.
Does NTA’s Flow Storage Database work with SolarWinds EoC
- NTA 4.0 is supported by SolarWinds EOC – Enterprise Operations Console. (no change compared to NTA 3.x)
How do I set up redundancy for the NTA Flow Storage database Server?
- There are no specific requirements for ensuring redundancy of your NTA Flow Storage; you can use a distributed file system, leverage network attached storage, or use VM Ware High Availability. If redundancy is required for your NTA Flow Storage, please check with your server or systems team on how to best solve this for your specific environment.
SolarWinds' FoE (Fail Over Engine) does not handle the NTA databases (neither SQL nor FSDB). However, NTA 4.0 provides a mean to manually or periodically (scheduled) backup the FSDB database.
My DBA wants to know more about this Flow Storage Database, how does it work?
- The NTA Flow Storage Database is a columnar oriented database that leverages a very fast bitmap indexing technology. Data is stored in partitions that are created per few sources per day or per max size. New partitions are created daily or as necessary.
How do I access this data? I used to write SQL Queries to data mine the flow data in SQL
- Now that the flow data is being stored in our own database you can directly query the raw flow data by using the SolarWinds API which leverages our own query language, aptly titled SolarWindsQueryLanguage (SWQL)
I had written some custom reports using SQL, will those still work?
- Old custom reports have been archived for you to reference, you will need to rebuild those reports using the new web reporting interface that ships with 4.0. If you are unable to write a similar report with the new web reporting, you can always use the SolarWinds API/SWQL
How does archive and restore work with new Flow Storage Database
- Archive and restore of SQL DB has not changed, to backup and restore flow data you have to use backup and restore functions of Flow Storage DB.
Any differences in the way CBQoS data is handled?
- There is no fundamental change in the way CBQoS data is handled, between 3.x and 4.0: In both, the default polling period is 300 sec, and the CBQoS data is retained for the configured retention period.
The difference is in the displaying, because NTA 3.x was aggregating the CBQoS data consistently with the flow data, in order to provide chart consistency.
In NTA 4.0 this CBQoS aggregation has been removed in order to show as-polled data. It will therefore show more granular data than 3.x, even if the polling and retention activity remains the same as in 3.x (no dabase sizing difference due to CBQoS between 3.x and 4.0)
FAQ on Migrating Data from SQL to the new Flow Storage Database:
CanI migrate my historical Data from the SQL Flow Storage?
- Yes during the configuration wizard you can choose to migrate the historical data from SQL, this will run in parallel with new flow storage until the migration has completed.
Will my historical data from SQL be cleaned up / deleted
- Yes, as the data is migrated to the new Flow Storage Database it is deleted from the SQL database
Do I have to migrate my data from SQL to the new Flow Storage Database?
- No, you have two additional options surrounding migration. You can choose to not migrate and leave the data in the SQL DB (however this data will not be accessible from NTA after upgrade, but you can still access them directly in SQL or using dedicated evaluation NPM+NTA installation), or you can choose to not migrate and clean the data out of the SQL DB.
Can I capture new flows while migration is happening?
- Yes, the migration will run in the background until complete. The application will automatically load balance the two operations (recording flows / migrating)
FAQ on Installing a Release Candidate - RC - version of NTA 4.0
Does the RC need to be applied to the database server as well as Orion and Polling engines?
- Whether you apply an NTA 4.0 RC to NTA 3.x or to a previous NTA 4.0 RC (e.g. RC2 over RC1), the RC needs to be applied to your entire NTA environment, i.e. main poller, flow storage (only for the case of RC2 applied over RC1 ), all additional pollers and all additional websites.
I am experiencing a failure of NTA Flow Storage Configurator, what happens?
- It is a known issue of RC2, which wil be adressed in a further release. For now, Cancel it, and start the NetFlowStorageServer service manually. It just takes longer for the service to start.