Post the worst server setup you ever used/saw/heard of that was used for something else than testing.
>tfw you realize some people use machines without ECC memory as servers
I don't have any pictures of servers, but I remember contracting for IT a while back and replacing an IT guy who didn't know what the halt on keyboard error meant on a server.
He said that he "didn't work on servers".
Yeah it was a trip.
I guess he inherited the network from some other admin. Must have spent his days doing virus scans on workstations and reinstalling adobe reader.
The place was a total clusterfuck with the proverbial server room / cleaning closet, and a snarl of CAT5 plugged into some ancient switch. Total fucking mess.
It was a non profit humane society, so I had to fight tooth and nail to get them to buy any decent equipment.
I did end up getting them a couple of Dell servers for file hosting and exchange, and some Dell workstations which was a major improvement.
Unfortunately, they turned over people like crazy, and the place got thrashed.
I did make it a point to check out their excel spreadsheet of payroll, and then it all made sense. Employees had like 20% of payroll, and the director was getting the rest, but they normally just used volunteers.
What a joke.
i liked my hobo build when i was using it. complete with a hobo picture
4 servers, 4 and a half years, not a single problem, no downtime. Going half a year to a full year in between reboots.
I7 2600K's, gigabyte consumer mobo's, Kingston valueram and Crucial M4 consumer SSD's.
>where one lives depend on working computers ecc is essential
How often does an error occur in memory? How hard is it to predict statistically how many memory errors you'll have. Is it a function of amount of memory * time? Is it a function of writes? Is it a function of reads? To the best of my knowledge a memory error has never occurred on any computer I've used.
sounds like something you would want to get out of ASAP.
This really isn't the worst SERVER setup per say, but definitely the worst network setup I've ever seen. My Dad runs a small business of about 20 employees, and had a part-time IT managing it all. When he got fired my Dad asked me to come in, check everything over, and help him hire a new IT. So I agree to come in and take a look. I spent 5 hours driving back to my hometown to a fucking disaster.
The main DC/DHCP/DNS/shared storage box I setup years prior wasn't connected. He disconnected it, swapped everyone back to local domain, and he had gotten a consumer grade mid-end router to manage all network traffic and DHCP. So for god knows how long, a massive energy-hog of a early 2000s server was just sitting powered-on and doing ABSOLUTELY NOTHING.
The other employees complained that they had lost access to the network drive ages ago, complained, and never got a clear answer or solution as to why they couldn't access it. (I wonder why....)
The switch rack was a total mess with spaghetti cabling everywhere. He had taken half of the lines out of the patch panel, stuck on connectors (MFW), and wired it directly to whatever random connector the switch had open. That panel was setup when the first cables were ran to label where they went, and I got to figure out what went where again (MFW, again).
I just wish I could have talked to the guy to figure out what the fuck he was thinking.
My school uses netware for the email servers.
Teachers have a nice one-part domain name and can configure their devices to the server.
I've pretty much abandoned my school email and i think i'll use some shady, self-provided email address instead.
>not a single problem
Except for the invisible bit rot in all your data.
Yes, but somehow i'm afraid of breaking it. I'm still waiting to RAMdisk because the program make BSoD my PC three times. I'm afraid to do shit i can't fix. This doesn't happen on my laptops...
>Is it a function of amount of memory * time?
It most strongly correlate with altitude (!) and time. Research has shown that the majority of one-off soft errors in DRAM chips occur as a result of background radiation which may change the contents of one or more memory cells or interfere with the circuitry used to read/write them. Electrical or magnetic interference inside a computer system can cause a single bit of DRAM to spontaneously flip to the opposite state. Error rates increase rapidly with rising altitude. Compared to the sea level, the rate of neutron flux is 3.5 times higher at 1.5 km and 300 times higher at 10–12 km (the cruising altitude of commercial airplanes).
A very large-scale study based on Google's very large number of servers was presented at the SIGMETRICS/Performance’09 conference. The actual error rate found was several orders of magnitude higher than previous small-scale or laboratory studies, with 25,000 to 70,000 errors per billion device hours per megabit, i.e. about 5 single bit errors in 8 Gigabytes of RAM per hour using the top-end error rate, and more than 8% of DIMM memory modules affected by errors per year.
There is no way to detect these errors in non-ecc ram, they are completely silent.
>not using btrfs
Using btrfs without ECC ram
>but ecc boards are expensive
>$100 extra to guarantee the integrity of your priceless data that you intend to keep for decades
>5 single bit errors in 8 Gigabytes of RAM per hour
I'm running our business's (only) server (Dual Xeon 2660 and 64GB RAM) on a consumer ATX PSU and Samsung consumer SSD's in RAID 10.
Also the server is hosting a Employee time tracking software and ERP, both written by me, along with Terminals built on Raspberry Pis.
Oh and the whole thing runs on Debian and Ruby.
When i quit, this whole thing will one day go up in flames.
>Every time that data is copied to ram, single and double bit errors are introduced. If it's written back to disk again (as in say, making a backup), the data now has invisible errors in it..
>don't put all your music, movies and documents inside a 128GB ramdisk
>Implying data magically appears on HDD platters without going through ram
>When i quit, this whole thing will one day go up in flames
They should fire you already for not using industry standards.
Did you take the decisions on how to implement the solutions or was it your idiot boss?
If it was the latter they should fire him instead.
>It most strongly correlate with altitude
lol altitude. I makes some sense though. 5 signle bit errors per 8GB per hour seems really high. If windows 8 takes up .5GB of RAM in my laptop with purely instructions then my OS should have 7.5 bit errors per 24 hours. So after running it for a week without reboot it should have 52.5 bit errors. Since the likelihood of them occurring in the same instruction is negligible that's 52 bad instructions. Shouldn't this cause a crash? Is there some other kind of error correcting going on here? Is there some kind of redundancy I'm missing?
>does less ram mean only fewer errors?
Yes and no. 4GB ram will get half the number of errors 8GB will get during the same time frame. But if copying a file takes 10MB ram, you'll have the same risk of errors happening to that data transfer no matter if you have 4 or 8GB total ram.
> implying this isn't a normal process ANY computer goes through
so it's more dangerous for huge files, say movies (20 GB), to be copied from a desktop to an external hdd (8 GB ram, 5 bit flips per hour), copying takes some time --> bits flipped?
Any data gets corruped all the time, so from the past (Vinyl records are worn out at every playback) to the present nothing has changed (data gets worn out every time copied).
Nobody has ecc ram in the desktop machine?
>sounds like something you would want to get out of ASAP.
Eventually they fired me due to lack of funds (kek), but they never did remove the installation of SSH that I used to pipe VNC through...
It's like the guy had never heard of google. He just put blinders on when he was confronted with something he had no experience with.
All of that shit is my doing and it just werks perfectly, who needs Industry standards if you can get some faggot like me to write a complete ERP for less than the server hosting it costs
And also since the atmosphere is capable of acting as a shield according to your data, then the earth should be much more of a shield. So orienting your chips so that the flat side of the die is facing the sky would cause more particles to hit it right? But then again rays striking the chip parallel to the face of the die would pass through more transistors. BRB digging a hole in the ground and ordering lead plates.
How hard is it to detect these errors? Is it as easy as I'd imagine it? I'm thinking of writing a program that allocates 2 1GB chunks of data and then assigning a very precise binary representation of a constant like pi to it. Then XOR the data every fifteen minutes and count how many bits of the output of the XOR are 1. If the 5 errors per 8GB per hour rate is accurate, then that program should detect 1.25 errors an hour. But then there's the question of what kind of correcting is done. Is there any error checking or redundancy somewhere that would fuck it up? It's kind of neat thinking of my computer as a particle detector
In this paper, we analyze meas ure- ments of memory errors in a large fleet of commodity servers ov er a period of 2.5 years. The collected data covers multiple vendors, DRAM capacities and technologies, and comprises many millions of DIMM days. The goal of this paper is to answer questions such as the following: How common are memory errors in practice? What are their statistical properties? How are they affected by external factors, such as temperature and utilization, and by chip-specific factors, such as chip density, memory technology and DIMM age? We find that DRAM error behavior in the field differs in many key aspects from commonly held assumptions. For example, we observe DRAM error rates that are orders of magnitude higher than previously reported, with 25,000 to 70,000 errors per billion device hours per Mbit and more than 8% of DIMMs affected by errors per year. We provide strong evidence that memory errors are dominated by hard errors, rather than soft errors, which previous work suspects to be the dominant error mode. We find that temperature, known to strongly impact DIMM error rates in lab conditions, has a surprisingly small effect on error behavior in the field, when taking all other factors into account. Finally, unlike commonly feared, we don’t observe any indication that newer generations of DIMMs have worse error behavior.
>If stored correctly
Storing it correctly is costly. You have to control the humidity and temperature to very strict tolerances, any sudden fluctuations can quickly fuck up the tape. Especially high moisture will kill a tape within hours.
>So orienting your chips so that the flat side of the die is facing the sky would cause more particles to hit it right?
I suppose so.. kek
How hard is it to detect these errors? (...)
Such a program would be very interesting. Surely it already exist? Makes me wonder why studies haven't been done like that, just monitor errors in PCs in the wild.. There has to be a caveat to that method.
>do it slowly ?
Do what slowly? Actually if you suck out the air so it's severely under-pressurized, temp fluctuations wouldn't affect the internal pressure that much. Temp fluctuations would still be an issue tho, if fucks with the binder material.
I used a old laptop to host my homepage on the Internet. Old hardware and no security whatsoever apart from default on Windows 7 or vista... I can't remember... And it was my daily computer at the time as well, so it hardly functioned as a server.
this one is a bit better as it uses oil to cool
>Accidentally knock the cup of water over the motherboard
Should work well, but since you need to be able to open and close the box, you'll have a seam that'll be hard to seal. And maintaining the vacuum passively for a long time is a challenge.. Heat sealing a vacuumized plastic wrapper around the tapes, and then storing them inside an actively maintained vacuum box should do the trick. If you wrap that entire setup inside some thick insulation you should be set.
It's not really a "server" per say but it runs some IRC channels/bots and works as a torrent machine too.
this is better than buying ecc ram, because even with ecc ram you could still suffer from harddrive failure (controller up fo headcrash), and then what?
But muh fancy ecc reduces errors, no?
You can't correct what is lost.
what's the worst thing that could happen with those 'invisible' ram errors introduced to files
a single fixel out of color in a movie? a letter changed in an ebook?
or would it just plain corrupt it and make it unreadable? if so i'd rather have a back-up/raid server then ecc ram, although ecc ram is very tempting it's not primordial
>this is better than buying ecc ram
If you transfer your data to tapes on a machine without ECC ram, you're just as fucked. At least with zfs + ecc your data is continually monitored and any errors discovered and corrected. With tape you'd have to run a complete integrity check every time you copied the files back.
A ship spends 4 months at sea doing seismic testing.
1 bit is bumped out of place.
400+ Dual gpu servers begin to render the effect of each angle of each reflection. Counting historical refrences to see any mightations in the data.
An anomily is detected and a new well is drilled in the area identified.
approximatley 2 million dollars has just been spent to drill a hole in the dirt. Nothing is at the bottom of the hole. No oil.
>what's the worst thing that could happen with those 'invisible' ram errors introduced to files
A single decimal changed from 1 to 0 in your rocket_engine_specifications.xls that cause the entire rocked to explode catastrophically on launch killing all the astronauts on board.
..then the resulting fire spreads to a nearby jet fuel refinery which blows up setting of a seismic event interpreted by the pentagon as a nuclear attack on US soil.. ..Which initiates a full scale nuclear retaliation on all nuclear equipped countries, which in turn sets of the second-strike capability of Russia and marking the beginning of WW3 and the extinction of modern civilization and eventually all human life on earth.
That's about the worst that could happen.
Don't forget the runaway electrons issue, as the circuitry degrades over time so does it's ability to contain high energy particles on power off/power up, increasing data loss and amounts of harmful radiation released to the sorroundings.
Value your data and more importantly your health
Don't cheap out on a mobo
im sorry i was referring to a not so scientific/professional ambient but i know the implications of a single number changed could be catastrophical when you are dealing with exacts but as far as 'home' use thats what concerns me thats why i said movies and books
Worst case scenario a single flipped bit could corrupt an important file. Most uncompressed data will be fine, as long as it doesn't happen in the header. Compressed data on the other hand doesn't like it at all. A jpg can be corrupted by a single bit error, same with mp3, archived data, executables etc etc. Text and numbers, like spreadsheets, documents etc also doesn't like it at all.
>well i have backups so if anything gets corrupted im gonna be fine
That's fine and all, but it won't protect you from future errors. Say you make a picture in PS, then just before you save the image a bit flip happens in ram. Hey presto, you've just written the corrupted image to disk. Same applies if you download a file etc. Unless you manually check each file after it's written to disk and immediately compute a hash for it, you're not safe. And even if you do all that, the hash might be corrupted unless you use ECC, and written to disk in a corrupted state.
Home media plex server which fucking version I'm a god damn moron
the only way you'll learn to fix something is by fixing it when it breaks.
get a spare hard drive, copy everything you care about to it, and get experimenting.
There is literally nothing you can do to damage your PC by just re/installing software.
I actually have debian currently installed on it, but for some incomprehensible reason it won't let me do the two things I want to do on it:
>set up remote desktop ssh to access it from my Windows gaymen PC for convenience.
I don't know a thing about debian/linux, but I'm following the instructions to the exact letter to no avail. I figured I must have installed it wrong.
yeah, who ever would put a RAID of SSDs inside his server and save all store all his raw footage for his tech tips youtube channel on it
that would be so stupid
well maybe you're right, in the long run ecc is gonna be always superior and less time-consuming then non-ecc ram but what the hell man im just a home user im not a hacker, my files aren't so valuable that a single bit would result in critical damage
i never said i didnt care, and even if i didnt care it doesnt mean they arent valuable to me
my hair is valuable to me but i know its gonna fall some day when im old so i dont care what happens to it
>my files aren't so valuable that a single bit would result in critical damage
I guess it just depends on what data you have. I store a lot of irreplaceable data, such as scans and images,and many years worth of my own work and project files. Most of my stuff will only get more valuable to me with time, while at the same time the risk of data loss increase with time. I intend to pass on a lot of my data to my kids one day, which means my perspective is decades rather than months or years. Maintaining data integrity for that kind of time frames requires a little more than burning a few DVDs and calling it a day. To add to the hassle, I constantly add new data to this collection as I digitize more stuff, and I also work with a lot of the raw data in post processing. For me the solution is UPS + ECC + ZFS + hashes of each file saved on a NAS as well as on on site and off site HDD backups + one encrypted backup in the cloud.
Nabbed this pup the other week
> SuperMicro X8DTL-i >Two Xeon E5506
all for $158+shipping
i know it can but most of my most cherishable possessions i have them in physical form so assuming anything gets corrupted here i could always get it back
only thing that worries me is literal rotting but everything's saved up nicely here
i dont have photos i dont like photos and while i know they lose colour i think its part of their charm
if anything i could always keep the negatives around saved up nicely in a dark place and out of humidity and air, hell i could put them in a bag and save them in my fridge too
>start sysadmin job at company
>old admin wasn't a complete moron but a few things could be cleaned up a bit
>drywall and construction dust and crap all over server room from some project years earlier
>about a month in someone complains that the Lotus Notes server is down
>this is the first I hear of even having a lotus notes server
>look around for it in the server room, there's definitely no lotus notes server
>find this supposed lotus notes server IP and check the arp table
>locate port on the switch and start tracing the cable
>it leads to the corner of the room and UNDER the fucking wall
>ask boss what work was done in the server room
>oh not much we had to put a new wall in
>mfw there is nothing on the other side of that wall except for about 1' of space, some pipes, wiring, and apparently a whitebox Lotus Notes server
>mfw they actually didn't realize they were building a wall around a fucking tower PC
Drywall server guy here got another one but dodged a bullet with it
>looking at starting a short term (<1yr) contract with company
>their sysadmin was some old fart who was probably hot-shit in the 90's but not anymore
>I ask to see their infrastructure before signing on
>open server room door only to discover things that nightmares are made of
>for reference this was in 2009
>10BaseT hubs EVERYWHERE - I'm talking like daisy chained, big ass 24 port hubs running EVERYTHING
>CAT4 cable mess that would make a tentacle monster blush
>no AC units at all literally no cooling system in a room with 4 filled racks
>whitebox servers scattered everywhere, they're stacked on the floor and RUNNING
>AN ACTUAL LEANING TOWER ABOUT 7' HIGH OF RUNNING WHITEBOX SERVERS
>oh and by the way your desk is in here as well
NOPE NOPE NOPE
2011 first gen i5, 4 gigs of ram, a hard drive from a company in korea I've never heard of and that HP quality... Runs Ubuntu for a couple game servers and torrenting. Backups are done wheneverthefuck with an external drive. Coupled to a shitty netgear router behind the steel plate to muffle the insane coil whine.
This is my home server in my HoboRack™
I quite like it.
>tfw you try to build an ECC workstation but find it almost impossible to find a decent combo of compatible components unless it's an overpriced dual ATX Xeon server board monstrosity