The Securities and Exchange Commission is starting to get a cloudy outlook on business. Over the last three years, the organization has taken steps to upgrade its technology infrastructure and has recently begun moving programs and operations to the cloud.

The agency has transitioned it's main operating functions to a cloud platform, and will begin moving more minor operations in the coming months. According to SEC CIO Thomas Bayer, the agency has seen improved speed and flexibility in its financial data services, and boosted efficiency has dramatically cut costs; the commission saved $18.5 million in operations and maintenance expenses last year.

Increased benefits with dedicated solutions
The SEC manages immense amounts of data to provide the financial industry with analysis and updated information, causing a great strain on the agency's website and data centers. To improve uptime, the commission reduced its data center presence to just two facilities supporting a private cloud. Space, power and cooling services are leased from an outside vendor. According to Bayer, the transition has saved on operating costs and greatly improved the availability of the agency's website.

"We realized we needed this data all the time, not just in an emergency, to understand detailed patterns, monitor the markets and inform us on how orders flow, interactions with trades, rates of cancelations, volatility and other things that are really central to understanding market structure," said Gregg Berman, director of the Office of Analytics and Research in the SEC's Division of Trading and Markets.

Putting big data to use in the cloud
In an effort to fully harness the power of their cloud platform, the SEC has also begun a big data initiative known as the Market Information Data Analytics System. The MIDAS program is being used to provide rapid analysis on financial trends to organizations and customers visiting the agency's website. A filing by a major company can cause a massive rush of users requesting information.

"You can get 88 million hits in an hour and be able to serve that up [with human analysis],"
Bayer told InformationWeek.

MIDAS isn't only being used to understand market trends or rapidly emerging trading methods, but to help inform the SEC on future policy making. More than 400 gigabytes of market data are collected by the agency on a daily basis that can be used to better understand how the market works. As it comes, the MIDAS data archive would contain about 1 petabyte of data. Compressed, the system holds just over 100 terabytes.

"The downpour of data generated by the markets every hour will lead to better regulation and better investor protection," said SEC chairman Elisse Walter, adding that MIDAS will "dramatically improve our understanding of the way today's markets function."

The SEC decided to make MIDAS a hosted system so the agency could focus solely on analyzing the massive amounts of data they were receiving instead of worrying about day-to-day maintenance.

Improved uptime, security with cloud
Plans are currently underway to move the entirety of's capabilities to the agency's private cloud, improving both security and functionality. One of the site's most important components, the Edgar electronic filing records system, causes a lot of strain on the website and processes need to be put in place to ensure redundancy. The site receives 2.5 petabytes of data daily and more than 2 billion hits a month, according to Bayer. The commission's private cloud platform is able to handle the large amounts of traffic, as well as surges the site experiences related to a filing by a major company or a rule change by the agency.

Once the SEC moved its website operations to the cloud, it noticed an improvement in its data refresh rates of more than 300 percent. Bayer noted that, with the cloud, they were essentially given unlimited bandwidth, making continuity of functions much simpler.

Adopting a private cloud solution has also greatly increased the security of the SEC's data stockpiles. Bayer noted that it was important for the commission's financial information to be protected, as they were regulating the security of other organization's data.