Page 1 of 3 123 LastLast
Results 1 to 10 of 23
Like Tree13Likes

Thread: Anime-Sharing Project Infrastructure: An Overview

  1. #1
    ピュア&イノセント Checkmate's Avatar
    Join Date
    Oct 2010
    Location
    Castle Groundgroup
    Posts
    6,627
    Thanks
    202
    Thanked 5,216 Times in 823 Posts
    Points: 151,034, Level: 100
    Points: 151,034, Level: 100
    Overall activity: 0%
    Overall activity: 0%
    Achievements:
    Three FriendsOverdriveRecommendation First ClassTagger Second ClassVeteran50000 Experience Points

    Post Anime-Sharing Project Infrastructure: An Overview

    Hi!

    Today, I'll be sharing some background details about the infrastructure behind AS Project. I think some of showed interest in that, and maybe a couple more will read it out of curiosity.

    AS Project Overview:

    AS Project is currently powered by Celato Security, a rather small cluster I assembled in late 2008, mainly for educational purposes. When I built the servers and created the first draft of the network behind ASP, I was merely a business student and a total idiot when it came to computers. Therefore, I needed to do a bit of research before AS Project could launch.

    Originally, when I first planned Celato Security, it consisted out of only three servers, each with the same specs. Those were Core 2 Duos 8400 with 2GB Ram and 500GB HDD space. They sat on a 100Mbit/s post and had a bandwidth cap of only 10TB. To avoid legal issues I decided to host outside of the US. Monthly cost was at $250.

    Planning for ASP continued to progress in early 2009. Before the project even left the planning stage, a major revision was made. I added a distribution server, so we can increase the amount of content provided. It was a Quad Core Q6400 with 4GB RAM and two 500GB HDDs. It was on another 100Mbit/s port with a ridiculously small bandwidth cap of 3TB. Nothing fancy, but it would become instrumental in our continued success. This server increased monthly costs by $130.

    During 2009 and early 2010, I gained plenty of crucial experience about designing effective systems and making them work together. I also learned which combination of hardware would be best for a specific task.

    Back then we started to host our images on our own, private server. Soon hotlinking became a problem: Thousands of people hotlinked from our personal image repository because of the low loading times and permanent retention. This added immense stress on our wallet - around $350 every month had to be paid for other people's greed and laziness. We quickly came up with an effective counter tactic. We started watermarking all of our boxshots and screenshots, which impeded the bandwidth theft somewhat and kept the project affordable.

    All the time the cluster underwent upgrades before it finally ended at its current stage.

    Current Setup:
    • Cake: A Dual Processor Harpertown with 6GB RAM and two 1TB RE HDDs in RAID-1. Functions as webserver and has a dedicated 100Mbit/s line.
    • Finnel: A Single Processor Q8400 with 8GB RAM and four normal 2TB HDDs in RAID-0. Functions asdistribution server and has a dedicated 1Gbit/s line.
    • Cordelia: A Single Processor X3480 with 8GB RAM and two 1.5TB RE HDD. Functions as ASL Project server and has a dedicated 100Mbit/s line.
    • Yusa: A Single Processor, Q8400 with 8GB RAM, two 1.5TB RE HDD and a 120GB SSD. This server was donated for oreno.imouto.org - the world's favorite scan site. It has a dedicated 1Gbit/s line.


    This setup is backed by two outsourced offloading services, which provide DDOS protection and also work as CDNs to serve static files. Cloudflare is one of those two services.

    The whole network ran smoothly during 2010 and and most of 2011. However, at the end of the 2011 we experienced a major traffic surge overloading the whole setup. Traffic tripled in a short amount of time and reached a point where it was twice the amount we set as maximum for the current setup. Still, with extreme server optimizations we managed to serve 500 000 pageviews daily to over 2 000 000 monthly visitors. Given how standard usage for a webserver like this puts it at a mere 150 000 pageviews daily, this comes close to a miracle.

    Apart from the servers mentioned above, we also use several VPS for a score of minor tasks, services, and projects.

    Our original plan saw us upgrading our infrastructure somewhere between the second and third quarter of 2012. But with the still ongoing traffic increase we'd be pretty much dead till then. As Project Manager of ASP, I decided to expedite the upgrade and get it done during the fourth quarter of 2011. The upgrade will be a major one - nearly the complete Celato Security cluster will be replaced with more powerful machines, bringing us effectively to Celato Security Mark II. Tentative estimates put completion of the upgrades to early December.
    The only server kept is Cordelia since it was a recent acquisition and does not yet need replacement.

    Admittedly, I never in my life set up something as powerful as Celato Security Mark II.
    It already started with carrier selection. We not only want a premium bandwidth carrier providing high quality tier 1 backbones (Tinet, Level 3), but also one that is affordable. This is not an easy task as we're continuing to host outside of the U.S.A. This means bandwidth is at least five times more expensive than a state-side solution would be.

    The hardware selected was also chosen to serve us for a long time and on a major scale. After days of deliberation, the final setup was decided upon:
    • All current servers will be replaced with two new ones each. The new servers will have the latest Xeons available to us.
    • A global backup server will be added.



    Future Setup:

    Finnel's two replacement servers each have a Xeon E3 1230 with 8GB of ECC RAM and use a 1Gbit/s burstable line.

    This is an obvious increase in processing power, but one may wonder why the bandwidth downgrade. The answer is simple and consists out of two parts. On the old server bandwidth wasn't as much the problem as disk I/O was. Both a single server with fast enough disks and two smaller servers with dedicated 1Gbit/s line would be a lot more costly.
    Dedicated 1Gbit/s lines have always been very expensive, and two of them would've caused our operational expenses to skyrocket. Instead we opted to go with the a lot more affordable burstable option, which still gives us dedicated 100Mbit/s lines with a little extra from time to time.

    The switch actually occurred last week. So far, the huge amount of workload distributed nicely among the servers, and stress was reduced to near nothing. Overall, our experiences show we have approximately the same amount of usable bandwidth (at those speeds the servers from filehosts actually start to slow you down anyway) coupled with serious performance gains. The ancient Q8400 just can't compete with the new Xeons, the disks used are faster, and the new bandwidth cap is still generous enough with 30TB for each upload and download, per server.
    Given that our costs increased by a mere $10, the switch has just upsides. This means we can now upload even more anime, eroge, and hentai for you!

    Now, Cake's replacement will also consist out of two servers armed with Xeon E3 1230s. The RAM will be a lot higher, though, with 16GB EEC for each server. One of them will function as webserver, the other as database server.

    To my surprise, a lot of people asked why we'd go with two smaller servers instead of one big one. And why we didn't opt for a Dual Xeon. This made me seriously reconsider my setup, but eventually I decided staying with two smaller servers brings a lot of upsides over one big machine.

    There's quite a few advantages:

    1. Server applications that we are using don't scale well to 16 logical cores. Two Xeons on one server would've meant just that: 16 logical cores. Any cores over 12 are rarely used by the OS because of how thread handling was designed. Research also shows that currently 8 cores are pretty much the sweet spot for server performance, which is another argument for single Xeon setups. (Note: This claim might not be entirely true but it was true with our setup)
    2. Dual Xeon boards are only available with the old LGA1366 socket. This would mean we'd have to use the aging, second generation, 45nm Nehalem processors. While they cost approximately the same as newer generation processors, they have much lower clock speed (meaning less raw power) and lack the optimizations Intel introduced in later generation CPUs, specifically the second generation, 32nm Sandy Bridge processors used in conjunction with LGA1155.
    3. Webservers exhibit different behavior than database servers and thus need different parts. Combining both into one server limits hardware and software which can be used. It also drives the cost (for the same performance) up a lot. Jack-of-all-trades need more power than single-purpose machines and cost for faster components increases exponentially.
    4. Two servers offer higher redundancy. In the rare case of a server failure, the second server can still temporarily take over. While this reduces performance significantly until repairs have been done, it at least avoids a complete outage leaving people with unhappy "server down" notifications seen frequently on some less organized forums.

    There are some disadvantages, though:

    1. More servers need more space, which increases monthly costs.
    2. Since two servers use more parts, the failure rate increases. This is mostly offset by the redundancy included with having more than one server, though.
    3. Building two servers means more components, which translates to higher cost. This is somewhat offset by being able to use slower components, but I didn't do a full cost-comparison calculation since a single server setup has too many other advantages to make it viable just for a minor percentage in price reduction.
    4. More parts also means higher maintenance and replacement costs. Since server parts don't fall under a certain price, the slower parts used in a dual-server setup eventually may end up costing the same as the faster ones of a single-server setup.

    However, the advantages far outweigh the disadvantages. I don't know of a single competent administrator running a major forum from a single dedicated machine and that for good reason - it's not a good idea after all.

    Now, since I'm talking about hardware already, I may as well add in what HDDs are used in the different servers:

    The webserver comes with four Toshiba MBF2300RC in a RAID-5 configuration. These 10 000RPM disks give us an array providing 900GB storage space and fast access speeds. We considered some other disks as well (like the Seagate Momentus XT), but the high amount of caching provided by our CDNs to users all over the world mean these disks are pretty much the best available choice.

    And while 900GB of high performance storage should last us a long time, we can always use the new backup server to feed in more space. More about that server below.

    But first, the database server: It actually uses two different kinds of storage: Half Intel 320 SSDs and half trusty Seagate Savvio 143GB 15.2K RPM.
    This split is mostly caused by the fact SSDs are not designed for heavy writing duty. Therefore, the database log and buffer will be put on the Seagate HDDs instead since they are both write intensive (bad for SSDs) and sequential (meaning speed difference between SSDs and high performance HDDs is negligible).

    Those two servers and the backup system will be connected by a 1Gbit private LAN.

    The on-site backup server will have 40TB of usable space, where we'll backup our precious goods (anime and eroge, hourly database backup and webserver backups).The backup server will also run on a Xeon E3 and 16GB RAM. It will use ZFS filesystem. The drives used will be low performance, green drives since we don't expect high usage.

    Besides simple storage, the backup server has another important role: Should we ever need more hosting space (for whatever reason), we can always link it via iSCSI to our webserver. Don't get us wrong when we said it will use low performance disks. There'll be four 120GB SSDs used for caching which will provide great random read speeds and with 24 HDDs in the array, write speed should still be a lot faster than what you'd see in a desktop PC, enough to max out a 1Gbit line which is the bottle neck of the entire cluster.

    To keep things stable, a powerful firewall using deep packet inspection and advanced QoS will be between the Celato Security Mark II cluster and the rest of the world. As SQL database we'll actually be using MariaDB, which should provide even better performance and stability than MySQL.

    All together the cluster will cost around $8 000 to set up and a monthly $330 for bandwidth and colocation space. This upgrade should enable us to serve 10 - 15 million visitors every month. It will be quite a bit before we have to go back to the design table, meaning we can provide stability and speed for a long, long time.

    Hardware Overview:

    Webserver:
    Supermicro Chassis 1017C-TF Intel C202 1U
    Xeon E3 1230
    16GB ECC RAM
    4 x Toshiba 10K 300GB MBF2300RC
    Adaptec RAID 6805

    Database Server:
    Supermicro Chassis 1017C-TF Intel C202 1U
    Xeon E3 1230
    16GB ECC RAM
    2 x Intel 320 Series 120GB SSD
    2 x Seagate Savvio 15.2K 146GB
    LSI 9261 Megaraid w/ FastPath & BBU

    Backup Server:
    Supermicro Chassis SC846E2-R900B 4U w/ X9SCL-F C202
    Xeon E3 1230
    16GB ECC RAM
    24 x 2TB Seagate Barracuda Green
    4 x Corsair Force SSD 120GB Cached
    Intel SASUC8I SAS Initiator

    Projected Expenditures:

    Initial Investment: 7982$
    Progressive Upgrade: 2004$ by end of 2012
    Equipment Maintenance: 50$ per month
    Operational Expenditure: 785$ per month

    Expenditure by the end of 2012:
    Infrastructure: 9986$
    Operational Expenditure: 9420$

    Expenditure for 2012 - 2013:
    Infrastructure: 600$
    Operational Expenditure: 9420$

    Total expenditure from 2012 to fiscal year 2014: 38846$
    Currency: USD
    Error Margin: +/- 5%

    I welcome any feedbacks, suggestions, or comments about our setup.
    AzuAzu, SMSKA, Takami and 6 others like this.

    I only post what I bought. My uploads are for those who share the same taste but could not easily import them. If you can buy the game, please do it to support the company.
    Bugs report belong to the Site Support, not my inbox.

  2. The Following User Says Thank You to Checkmate For This Useful Post:


  3. # ADS
    Circuit advertisement
    Join Date
    Always
    Location
    Advertising world
    Posts
    Many
     

  4. #2
    ピュア&イノセント Checkmate's Avatar
    Join Date
    Oct 2010
    Location
    Castle Groundgroup
    Posts
    6,627
    Thanks
    202
    Thanked 5,216 Times in 823 Posts
    Points: 151,034, Level: 100
    Points: 151,034, Level: 100
    Overall activity: 0%
    Overall activity: 0%
    Achievements:
    Three FriendsOverdriveRecommendation First ClassTagger Second ClassVeteran50000 Experience Points

    Re: Anime-Sharing Project Infrastructure: An Overview

    The Xeon E3 Processors:





    HDDs & SSDs

    Intel 320 Series



    Toshiba MBF2300RC 10K RPM 300GB



    Seagate Savvio 15.2K RPM 146GB



    Chunk of Seagate Savvio 15.2K 146GB



    RAM and Raid controllers

    Kingston ECC RAM 8GB



    Adaptec Raid Controller 6805



    LSI MegaRaid 9260i w/ FastPath physical key and BBU



    1U interior:



    1U exterior (both):



    4U interior:





    4U exterior:



    2U interior:



    2U exterior:

    AzuAzu, Second_Flight and Dragon55 like this.

    I only post what I bought. My uploads are for those who share the same taste but could not easily import them. If you can buy the game, please do it to support the company.
    Bugs report belong to the Site Support, not my inbox.

  5. #3
    King of Hentai Sugre's Avatar
    Join Date
    Nov 2011
    Location
    Paradise
    Posts
    10,764
    Thanks
    1
    Thanked 895 Times in 858 Posts
    Points: 465,469, Level: 100
    Points: 465,469, Level: 100
    Overall activity: 0%
    Overall activity: 0%
    Achievements:
    Three Friends50000 Experience PointsRecommendation Second ClassVeteran

    Re: Anime-Sharing Project Infrastructure: An Overview

    The current setup is impressive and the plans are very promising. I`m amazed at how much money you need to have to maintenance this site. I think that as so soon you will become bigger community. A very big advantage is that on the Anime-Sharing has no ads.

  6. #4
    がはははー! AzuAzu's Avatar
    Join Date
    Sep 2011
    Location
    Sydney, Australia
    Age
    28
    Posts
    1,559
    Thanks
    11
    Thanked 402 Times in 246 Posts
    Points: 31,331, Level: 100
    Points: 31,331, Level: 100
    Overall activity: 0%
    Overall activity: 0%
    Achievements:
    Three FriendsYour first GroupVeteran25000 Experience Points

    Re: Anime-Sharing Project Infrastructure: An Overview

    Very sexy future setup, I'm glad you're going with Supermicro :>

    What's going to happen to your old servers :O I assume you're keeping Cordelia as it is? The Lynnfield Xeon is still very powerful ;x

    Oh, and since the expenditure is going up so much, is there any plans for a donation page soon? I know a lot of people are against that kind of stuff but you can sorta make it hidden or something~ I'm sure a lot of us will be willing to help.

    Thanks for letting us know whats going on, and ofc for those sexy new servers <3

  7. #5
    ピュア&イノセント Checkmate's Avatar
    Join Date
    Oct 2010
    Location
    Castle Groundgroup
    Posts
    6,627
    Thanks
    202
    Thanked 5,216 Times in 823 Posts
    Points: 151,034, Level: 100
    Points: 151,034, Level: 100
    Overall activity: 0%
    Overall activity: 0%
    Achievements:
    Three FriendsOverdriveRecommendation First ClassTagger Second ClassVeteran50000 Experience Points

    Re: Anime-Sharing Project Infrastructure: An Overview

    Quote Originally Posted by AzuAzu View Post
    What's going to happen to your old servers :O I assume you're keeping Cordelia as it is?
    Deprecated, they have little to no value being too old. Cordelia is just recently commission in May 2011, when we has a massive grand launch, it will stay for a while.

    Quote Originally Posted by AzuAzu View Post
    Oh, and since the expenditure is going up so much, is there any plans for a donation page soon?
    The only thing probably the initial investment, I managed to keep the operational costs increased about 150$/month for much better network and overall infrastructure.

    Quote Originally Posted by AzuAzu View Post
    is there any plans for a donation page soon? I know a lot of people are against that kind of stuff but you can sorta make it hidden or something~
    I have no idea, we don't feel like having a donation page right now, and we don't know when we feel like, or forced to, but I'd say it takes a while.

    Quote Originally Posted by AzuAzu View Post
    I'm sure a lot of us will be willing to help.
    That's something we hope for but I will still have a plan B in case things go sour.

    Quote Originally Posted by AzuAzu View Post
    Thanks for letting us know whats going on, and ofc for those sexy new servers <3
    We have been quite opened before anyway

    I only post what I bought. My uploads are for those who share the same taste but could not easily import them. If you can buy the game, please do it to support the company.
    Bugs report belong to the Site Support, not my inbox.

  8. The Following User Says Thank You to Checkmate For This Useful Post:


  9. #6
    King of Hentai Sugre's Avatar
    Join Date
    Nov 2011
    Location
    Paradise
    Posts
    10,764
    Thanks
    1
    Thanked 895 Times in 858 Posts
    Points: 465,469, Level: 100
    Points: 465,469, Level: 100
    Overall activity: 0%
    Overall activity: 0%
    Achievements:
    Three Friends50000 Experience PointsRecommendation Second ClassVeteran

    Re: Anime-Sharing Project Infrastructure: An Overview

    Quote Originally Posted by Checkmate View Post
    I have no idea, we don't feel like having a donation page right now, and we don't know when we feel like, or forced to, but I'd say it takes a while.
    Already I have an idea how to do it, but I will not be posting public. Checkmate if you want to know what I mean write to me on PM

  10. #7
    Renaissance Guy nr345's Avatar
    Join Date
    Sep 2011
    Location
    In a castle made of sand.
    Posts
    959
    Thanks
    64
    Thanked 20 Times in 20 Posts
    Points: 6,861, Level: 54
    Points: 6,861, Level: 54
    Overall activity: 0%
    Overall activity: 0%
    Achievements:
    Three FriendsYour first GroupRecommendation Second Class5000 Experience PointsVeteran

    Re: Anime-Sharing Project Infrastructure: An Overview

    So, I can count on Anime-Sharing like forever, there will be no problem occurs in future and you all will share without stopping. I don't know how can I show you my appreciation, sir.

    And

    I see that you're using a domain called "animesharing.info" as a CDN. Why didn't you used "cdn.anime-sharing.com" for CDN services?

    And...

    Which version of vBulletin you are using right now?

    At last...

    You should change name of your directories like admincp and modcp to another one.
    Last edited by nr345; November 13th, 2011 at 03:38 PM.

  11. #8
    ピュア&イノセント Checkmate's Avatar
    Join Date
    Oct 2010
    Location
    Castle Groundgroup
    Posts
    6,627
    Thanks
    202
    Thanked 5,216 Times in 823 Posts
    Points: 151,034, Level: 100
    Points: 151,034, Level: 100
    Overall activity: 0%
    Overall activity: 0%
    Achievements:
    Three FriendsOverdriveRecommendation First ClassTagger Second ClassVeteran50000 Experience Points

    Re: Anime-Sharing Project Infrastructure: An Overview

    I see that you're using a domain called "animesharing.info" as a CDN. Why didn't you used "cdn.anime-sharing.com" for CDN services?
    static files should be served on a different domain to prevent a cookie being attached to it, which the browser will need to process, overall less than optimal browsing experience.

    We optimized every aspect, even little details.

    Which version of vBulletin you are using right now?
    We cannot disclose the version information for security reason since we have a [stable] upgrade policy.

    You should change name of your directories like admincp and modcp to another one.
    It is planned, on the to-do list. Although I do not believe it being a big deal.

    I only post what I bought. My uploads are for those who share the same taste but could not easily import them. If you can buy the game, please do it to support the company.
    Bugs report belong to the Site Support, not my inbox.

  12. The Following User Says Thank You to Checkmate For This Useful Post:


  13. #9
    Renaissance Guy nr345's Avatar
    Join Date
    Sep 2011
    Location
    In a castle made of sand.
    Posts
    959
    Thanks
    64
    Thanked 20 Times in 20 Posts
    Points: 6,861, Level: 54
    Points: 6,861, Level: 54
    Overall activity: 0%
    Overall activity: 0%
    Achievements:
    Three FriendsYour first GroupRecommendation Second Class5000 Experience PointsVeteran

    Re: Anime-Sharing Project Infrastructure: An Overview

    Thanks for the answers.

    It is planned, on the to-do list. Although I do not believe it being a big deal.
    That is, sir. I got my website hacked just because of this weird situation.

    Also I gotta ask: "Which SEO processor are you using. It's so good! Even for Google, that SEO doing its job very good.

    Meanwhile, you should check the uploading system. We (means users) can not reach for servers to reference files locally. Always seems there is an "Error".

    A little suggestion, you might be able to use "robots.txt" to disable bots like Google-bot for an common instances

  14. #10
    ピュア&イノセント Checkmate's Avatar
    Join Date
    Oct 2010
    Location
    Castle Groundgroup
    Posts
    6,627
    Thanks
    202
    Thanked 5,216 Times in 823 Posts
    Points: 151,034, Level: 100
    Points: 151,034, Level: 100
    Overall activity: 0%
    Overall activity: 0%
    Achievements:
    Three FriendsOverdriveRecommendation First ClassTagger Second ClassVeteran50000 Experience Points

    Re: Anime-Sharing Project Infrastructure: An Overview

    That is, sir. I got my website hacked just because of this weird situation.
    The backend is nothing like you see

    AS Main Server sit behind several cached servers, what you see is merely a bunch of cached, static html files generated by the cached server. The cached server generate the static html instantly on demand using the data from the AS Main Server. Even if they manage to hack the cached server, for example, uploading a backdoor, all they have the control is the cached server.

    Cached Server wiped its content within 4 hours, and there are more than 12 cached server in the world. So when user connect to AS, he simply connect to the nearest, fastest cached server of AS.

    The cached server also act as a DDOS shield. In case of DDOS, there's just little to no impact since they cannot possibly DDOS all of the cached servers, placing different region of the world at the same time.

    Nobody have access to AS Main Server without a matching encryption key. The encryption key based on the hardware signature of the PC which used to connect to the Main Server.

    TL;DR:
    While being small, we do not the belong to the same lolsecurity level of Sony and Valve.

    And of course, we will change the default control panel login folder soon in the next maintenance, which is slightly more than a week from now on.

    Also I gotta ask: "Which SEO processor are you using. It's so good! Even for Google, that SEO doing its job very good.
    I used to be a pioneer of SEO with 4 years of experience. So, it's mainly personal experience. And then, I don't remember any SEO software that can actually do its job well without an experienced webmaster.

    Meanwhile, you should check the uploading system. We (means users) can not reach for servers to reference files locally. Always seems there is an "Error".
    We don't have enough resources for user to upload images and other content onto the server itself. Though I might mis-understand this, can you tell me how to reproduce the "Error"

    A little suggestion, you might be able to use "robots.txt" to disable bots like Google-bot for an common instances
    We have it, and in fact, google-bot does not crawl static files from the CDN.

    I only post what I bought. My uploads are for those who share the same taste but could not easily import them. If you can buy the game, please do it to support the company.
    Bugs report belong to the Site Support, not my inbox.

  15. The Following User Says Thank You to Checkmate For This Useful Post:


Page 1 of 3 123 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Hello People of Anime Sharing
    By Binso in forum Introductions
    Replies: 11
    Last Post: January 15th, 2012, 05:09 AM
  2. Hello Anime-Sharing.com
    By faceless in forum Introductions
    Replies: 9
    Last Post: August 12th, 2011, 09:48 PM
  3. [LPW #12] Go Go Anime Sharing~~ \(^o^)
    By BlueMage in forum Spam Section
    Replies: 2410
    Last Post: July 9th, 2011, 03:47 PM
  4. Ossu! Anime-Sharing!
    By ryuuda in forum Introductions
    Replies: 6
    Last Post: December 22nd, 2010, 02:54 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •