A while back I was developing a workflow scheduler. However the run duration kept coming back negative. The runs kept finishing like 15 seconds before they started. Eventually I figured out why; the company had configured a policy to use their custom NTP server, but it was 16.7 seconds off. So my Windows laptop's time was in the future, but the WSL VM (which ran the program) used another NTP server and was correct. In the end I just added a bodge that made the program request the time from an external server on startup to calculate an offset.

I spent a disturbing amount of time figuring out why I didn't get a video output.

I had not configured DP on the monitor.

It wasn't dirt on my monitor, it was a weird unicode character

https://xkcd.com/2606/

3rd line support for a global IT brand you have heard of, on the kind of security product that people don't like to mess with too much once it's installed on their servers.

Bug had been in the tracker over 3 years when I joined, and had defeated anyone who tried it. Segfault that happened to only one customer, unpredictably, and only once or twice a year, and nobody could reproduce it.

After a few weeks of tracing the compiled code line-by-line I found a race condition in the teardown of an object. That's not the silly part.

I wrote the patch for the version of the code the customer was running and was hailed as a mighty hero and everyone on the bus clapped etc. Then I went to forward-port the patch to the newer versions of the program and... it was already fixed there. Some fucken doofus had either fixed the bug without realizing, or, fixed it and just skipped the standard backporting process that would normally patch all the other versions that were still in support.

Forgot to save. You ever spend 45 minutes looking for a bug you already fixed?

Or variations of this caused by a makefile not having correct dependencies set up, eg it points to dependent source files but not the headers so header updates get ignored during compiles. Made me add a "make clean" step to the debug process of "change that should have fixed the issue appears to be doing nothing at all" issues, or even just defaulting to rebuilding everything because test times can take hours or even days to get to the issue, so I don't want to waste that much time only to find out it's still running the old version.

Yep. Last time I used MS Visual Studio it was still all too eager to run an older binary, and I lot plenty of time debugging a binary that doesn't reflect the current code. And, even more recently, when testing requires you to remember to run separate routines to upload to a testing server and restart a service, it is really easy to forget to do that when you are bouncing between the code and the browser console to figure out what to change.

My system didnt boot for obscure reasons

Found out something had caused my nix store (packages) to explode in size. Chrooting didnt work as it was filled to the last byte. Deleted some files, then chrooted in and did a cleanup

One time I was struggling to understand why a code branch I wrote was being unexpectedly taken, and I found that I had written something like if pet == 'cat' || 'dog'. Since 'dog' was a string, the condition was always true. I laughed out loud at myself.

A static type system can be used to make this type of error occur at compile-time rather than runtime. But, not everyone wants to make that tradeoff and some even like the semantics applied to that kind of expression. || often gets tasked with handling null/undefined in JS, e.g.

Back when I was learning c++ on Windows, I naively jumped right into game dev (as one does) before understanding the purpose of a debugger. I spent 3 months chasing down a period that should have been a comma. Due to function overloading, this was not an error or warning. It's just silently calling the wrong function and crashing the program.

Thus began my distrust of magic compiler features. To save typing a handful of characters, languages add new ways to blow off your own leg. Every time you blow off your leg, the time you lose is orders of magnitude greater than just being explicit. Just read this thread for more examples. Today, I program almost exclusively in C. I have no desire to use more feature-rich/higher level languages for anything.

ECU (the computer in your car) only came online on the bus at every 3rd restart. No memory was written, yet the bastard somehow knew it anyway.

Turns out the transciever's enable pin was left unconfigured and supposedly some charge accumulated during two startups to set it.

Love this

Goddamn case sensitive POS JavaScript.

I can't quite remember what I was debugging exactly, some constant was different in runtime compared to what was in code. Someone added a leading 0 to a const int to make the number align with others in terms of character count just to make it pretty I guess. That is when I learned about octal literals in C. I think that was the only time I've seen one in many years and it was used unintentionally. To save you a search leading 0 in C makes number to be base 8 same as 0x makes it hex aka base 16, so that inconspicuous 01000 is not a 1000 it's a 512.

It's one of the most sneaky legacy foot-guns C (and C-like languages) have. I think we're decades removed from using octal everywhere, except for Linux filesystem flags (which the letter-equivalents should probably be used instead).

Haha, I always love that one when I debug some other codebase

This was dumb, and it was on me.

Couldn't figure out why two obscure software packages were erroring in a very particular way. Thinking there was no way in hell a search would help, I plugged it in Google. There was user on Spiceworks outlining my exact situation and fix. Who is this mad genius?!

It was me. I fixed it two years previously and went back to note so in the forum.

Would have been even more hilarious if your previous fix was hidden by a "nm, I figured it out" update.

The one and only denvercoder?

Internet Explorer didn't automatically have "console.log" defined until you opened the console window. So there was an error breaking the web page that would disappear when you tried to open the console to see the error. The error was that console.log wasn't defined. Fucking IE.

That would genuinely make me clock the fuck out for the day when I figured it out, jesus christ.

/thread, holy shit lmao

I fixed a 5 year old issue that the previous guy claimed was "unfixable". Turns out it was because someone copy-pasted a value from an Outlook email..and it had a single dash in it...which outlook converted to a double dash.

That's why I was so confused about the AI emdash connection. MS Office has been replacing dash with emdash for 20+ years.

I was confused too. But now I think it’s specifically em dashes in the middle of words, isn’t it? Like copy—paste instead of copy-paste?

I have had issues of having the same characters but in Unicode failing. But if I type it manually or paste it into Notepad++ forcing it to ASCII first, it works.

Person before flipped a bunch of if checks to put the value before the variable (if 6==x rather then if x==6). They messed up a bunch of the <= and >= checks by forgetting to reverse them. I improved performance and fixed numerous bugs by tracking those down.

Ah yes, yoda conditionals. Because guarding against accidentally typoing if(x=6) is worth destroying your code's readability.

This is not needed anymore thanks to compiler warnings

Tell that to my colleague who still insists on writing if conditions like that. Maybe it's just a PHP thing.

I think php had a linter for stuff like this. Defensive programming is good, but having tools to do the defending is better so I can use my brain power where it matters

I was hired to work on a CMS designed for schools. I was tasked with debugging.

I started off with the newest issues and worked my way back.

Well, there was a bug where teachers would upload a video but it wouldn't play on the page. It was plaguing a team of 8 for YEARS.

It was mentioned in our morning standup, and I decided to fast track it.

The dude who wrote the code was shrinking the display size in half: well how do you think that would work when a dimension of the video isn't divisible by 2?

So, remember, I am the new guy, quickly fixed it using an if/then and the % modulus operator, and lo and behold, the next morning standup I had to explain the fix in front of the project manager, his boss, and the entire team...

fucking awkward. "You see, there's this thing in division called a remainder." This was the first thing in the morning.

But because the bug had been so ever present, that's why the boss's boss attended the standup. God, I just wanted to die.

Ok, I understand the issue but I don't understand your embarrassment from it. You didn't write the initial bug.

Btw, a faster way to check if a value is even is to just do a bitwise AND on bit 0 (if it is an integer, though it needs to be an integer for modulus to work anyways).

In many languages it looks like:

if ( val & 1 ) { \ it is odd }

Division is a multi-cycle operation but bitwise comparisons are simple enough to be done in a single cycle, or could even be thrown on to instructions as a free extra that doesn't add any extra cycles (eg any instruction with a bit mask, though it depends on the implementation as to whether it's done for 0 extra cycles or more).

Doesn't really matter for a one off operation like halving the value of video display parameters, but if you have a loop that needs to check a lot of them, the bitwise and will pull ahead.

And if you're thinking "well we need to divide by 2 anyways and the modulus is available from that operation as long as the compiler sees it should save the result and the modulus", a bitwise shift right (non-circular) of 1 is also way faster than divide by 2.

Both of these can also be generalized for any power of 2, if necessary.

I don't think I understand what the issue was or what the fix was. Was it a language or system that cannot perform the division with truncation?

Let's say height and width are 320x214.

(these numbers are arbitrarily chosen)

The code was sizing it down by /2, so now it is 160 by 107.

Because the width being odd, it would display.

But:

if ($width%2) { $width = --$width; }

Hastily written out on the phone, but that was the fix. This was back with php 5.

Sorry, but this comment still doesn't make sense (or has typos).

You can't halve a pixel. Truncating would get around the issue but big issue is that you can't have a float as a video resolution. The browser should just truncate it regardless but the original code is flawed.

OPs fix essentially does the same thing.

For a strong typed language, an integer division should not cast to any other data type. Even assigning a float to an integer should not be allowed (that's why C is not really strongly typed). I think from the syntax it was PHP. If any of the dimensions were not divisible by 2, then the number changed from integer to float and it could cause an issue somewhere downstream the video processing pipeline.

I don't think it's reexporting a video or something, I think it's the embed bc it is a CMS. PHP 5 isn't that old so video in webpages isn't a novel idea.

But you're right. Can't halve a pixel.

Do note that PHP5 is old, predating HTML5. Browsers probably still struggle with malformed HTML and in the absence of any sane value, just crash. Also, who knows if it was even HTML. For all I know it could be a flash.

Fuck, I forgot about Flash. I didn't think it was that old. Odds are it's just some other internal video player tbh.

OP didn't say that the problem was with the change in type. I think the interesting part is the downstream issue: what failed when a float was given while an integer was expected?

Dude, this reminds me of my coworker who was setting up a network and was trying to figure out a subnet mask, and he said, "I am trying to find a subnet mask between /22 and /23."

He spent the entire day on it. I finally had to ask, and once he told me his goal, I of course asked, "how the fuck do you get half of a bit?"

Oh, the blood drained from his face. "Yeah, bro, that's CIDR."

How does this not make sense? I am spoon feeding this shit. Are you just vibe coding?

I think there are some versions of Python and/or PHP where the normal division operator silently promotes to IEEE floats, and you have to explicitly cast back to truncate/floor the value.

Back in the 90’s I had to debug a printer routine in a reporting tool. It would occasionally crash. Only on Wednesdays. Only on Wednesdays in September. Only on Wednesdays in September after September 9th.

Code was written in c.

What was the fix. I'm a dumb data scientist, I don't know C

c requires you to do things like define the maximum length of a string. In fact a string in c is literally defined as an array of characters. There are no protections in c, so you can define a string to be 10 characters long and then copy 20 characters to it. The result is most often a stack overflow that immediately crashes the program.

This was legacy code that was likely written in the early days of DOS before things like Microsoft Windows existed, so memory usage was still pretty important. Whoever wrote this code tried to calculate the maximum size of a string used to print a header in this reporting package as a way to optimize memory usage. But they were off by one character. So when the code created a header with a date containing the longest month name, the longest date name, and a two digit day of the month, then that string overflowed and crashed the program.

Might be the character count. Wednesday has many, September has many, after the 9th has two.
Program filling the date into a form field of width 1 less than was possible with actual dates.

this but string length according to OP.

As an even dumber data analyst, something about this comment makes me feel seen. Thank you. <3

two days for an = instead of an ==

"Yoda conditionals" are a must in languages where bindings/references are mutable by default. E.g.: "if (0 == size)" or "if (0x4269 == magic)". If you write '=' instead of '==' you get a compiler error, so always put the constant first.

Hope you never have to use a language that lets you redefine numbers.

Wasn't there a Python version where True and False were mutable by accident?

I tend to use languages where immutability is the default these days like Haskell, Idris, or Purescript. But, even when using Java and Javascript, I tossed around a lot more final / const than my peers. :)

Redefining the number 2 in Python

The biggest const-by-default lang is probably Rust now isn't it?

Maybe? I don't really keep up with lang popularity. Got through the Rust tutorial and did one project with it, but I honestly was missing HKTs and (mostly) isolating effects based on type signature. Of course, Rust may have grown those features since then, it was last decade when I played with it.

I once spent several hours debugging a performance issue in the Linux builds of a project. The service worked fine on Windows but couldn’t keep up with the traffic when deployed on Linux. Interestingly, CPU load was actually higher on Windows, and no sampling profiler showed any suspicious hot spots.

It turned out the problem was in the program’s main loop, which was OS-specific and called something like sleep(float seconds) (yes, I know…). The parameter was set to 1e-3 on Windows and 10e-3 on Linux. I stared at that line several times before I noticed the difference.


Another time, all services of a distributed system were regularly reporting a loss of communication at 4 AM. The servers were supposed to receive data continuously, so even a few seconds of silence was enough to raise an alert.

Half of the team investigated all kinds of possible locking scenarios (the applications were multi-threaded). Then we noticed that all the logs in the system (including the kernel’s) had the same gap in timestamps.

Then someone remembered, that the customer had a cron job which did some maintenance at night. They confirmed that it halted the entire VM with all services for about 5 seconds.

i was just a user, but i debugged a weird issue where one of those blue light reduction programs would start flickering and inverting day and night time during june.

turns out the dev had made the assumption that the sun always rises before it sets, and i was far enough north that the sun would start setting after midnight.

https://zainrizvi.io/blog/falsehoods-programmers-believe-about-time-zones/

https://youtu.be/-5wpm-gesOY

Such a classic.

haha, knew what this was going to be. Already came close to posting it a couple of other times in this thread. :)

Ooh, interesting one! 😄

That I'm not cut out for coding. lol

Whenever I book a flight with Air France, I have to enter our birth dates off by one day. We're in Canada, and the Air France servers are most likely in...France, or at least using a french time zone.

Anyways, I figure that someone decided to use some form of LocalTime midnight for the date. We're 6 or 7 hours later, so when it translates it to French time, it's a different day.

So, not a bug that I fixed, but one that I work around. It's been this way for years, too.

The good news is that with JavaScript now having adopted Temporal in the standards all of that should be in the past as soon as apps switch to it. (And when Apple decides to implement the damn thing)

This was on a Fortran project that had no encapsulation and way too many global variables.

Suddenly, one particular calculation was failing, and not in a floating point rounding error kind of way. After setting up several scenarios to generate various input values, I determined that Fortran's absolute value function was returning the wrong results.

I couldn't understand why, so I went to the Lead Architect. He poked at it for about four minutes and did some searches, and found where someone else had defined a global variable named ABS that was masking the built-in abs function. The function call argument [e.g. abs(3)] was being compiled as an array index on this ABS variable [which was not an array, so yay, undefined behaviour].

Was told to test a new database migration as they updated from mainframe to a current version of Oracle. First I was to test data integrity. Wrote a python script to pull from the old db and verify that record matched the new. Found sooooo many bugs, sent it to the dev. Half were real bugs, half were bugs with my script. Turns out they didn't actually want a 1:1 on the new db they were going to change things cuz they could. Lots of back and forth and by the end my python script essentially mirrored the production code.

Then they wanted me to load test the new system. I wrote a quick script to query N rows from the db with X threads and blast the new service. Didn't take long to write, presented my finished script and plan to pull in manual testers to also run my script as needed.

Dev was cocky because they'd setup auto scaling on AWS. Manager was concerned I'd taken time to write a script instead of just spamming the same message over and over with a tool like gatling.

First run I took down the system very quickly, like within a minute with just my laptop. Note, this was a system that was planned to handle 100s of millions of requests an hour. Dev was impressed, turned out to be a auto scaling misconfiguration. Second time it took a few mins but it died again. Config needed some more tweaks. Third time I took it down because of some queueing issue that raised it's ugly head. Fourth time I took it down they found a bug in the code. Fifth, Sixth, and Seventh time I needed the manual tester's help, but the devs were scratching their heads trying to figure out how I was taking them out in 5mins of sustained load. Took two teams of devs scratching their heads and several itterations through bringing the system down and even two separate devs walked through my code to see what on earth I was doing. To top it off it wouldn't just take down the microservice, it would crash our entire AWS infrastructure for the company. Not just our project but all our aws projects in the building went down.

They finally found the problem. The silly little problem. Remember that data I pulled from the db and blasted them with? Well, we hit a poision pill situation. One of the records turns out was a manual tester's favorite to use when running one of his tests. That one record had 30,000 attached sub-records. Everytime that tester ran that test case he just kept adding a sub-record to it. Dude didn't know he was slowly crafting such a massive data bomb all those years that would take down all our AWS servers.

They had also fixed the problem with one service crashing the others. And that my friends is why you never skimp on good automated testing. When it went live there was no major issues or outages, though we did let it soak in parallel with prod for weeks to make sure we trusted the results.

I found an issue in a software used by government employees.
A request would fail if the server and the client had different years in their system time which could happen if they were located in different time zones.
After a few calculations I asked: "Hey, what are the chances a government employee would do this important task between 6pm and midnight on December 31st?"
We decided not to prioritize the issue.

You realise that's going to end up in the launch process for a nuke somehow.

“Hey, what are the chances a government employee would do this important task between 6pm and midnight on December 31st?”

I guess "this important task" could refer to firing a nuke, which to be fair is unlikely to happen at that moment in time (or any for that matter)

Govt bureaucracy. It could be scheduling a janitor and still somehow end up in the checklist.

I've been a dev for years and had my fair share of sillyness.. can't recall which was the most silly, but one very recent one was trying to debug a weird error. So I opened the VS solution did a search for the string that spat out the error. Found it in a catch clause and stabbed that function with breakpoints, and went off to test.

Hours later (too many to confess) after trying everything, I found out why I couldn't get it to stop on the breakpoint. I had opened the wrong project, for the wrong app :O

It was an unlucky coincidence that it had the same literal error trace in code!

i was trying to figure out why ssh wasnt working only to realize I was using the wrong username. That took about 2 hours of time to realize as I was bashing my head at the wall while looking at the pubkey signature being the same on both machines.

They did a 400 table join to return ‘true’ to test to see if they connected to the database; it took 48 minutes to run. Printed out, it took up 38 sheets of paper.

I replaced it with a dozen sequence checks and individual select statements for each table; it took 3 seconds to run. And it could accurately identify which tables and columns are missing. Printed out, it fit on a single page.

IT department really wanted to use their snake oil antivirus enterprise super secure hackproof™ software everywhere.

So it was installed on the CI servers.

Fastforward a few months and we were wondering why one build agent was no longer picking up jobs.

After a day of debugging we found out that a certain snake oil sofware tampered with a single machine internal TCP conncetion and somehow fucked it up in a way that all packets in one direction just vanished.

Disabling the snake oil software magically fixed the problem.

Should a CI server even have access to the Internet and need antivirus software? Maven access I guess.

It is good to have a build method that doesn't require network access; at one time that used to be required for Debian builds. However, for a variety of reasons, most CI servers do have access to the network, even if there's a firewall (or NAT) between them and the Internet.

Wrote 1 instead of i in a for loop and wasn't allowed to use a nice font like hack

GDB started printing emojis with MI mode: https://sourceware.org/bugzilla/show_bug.cgi?id=34501

"Yo, I hear you like emojis, so here are some emojis to debug, while you are debugging the debugger."

(I'm exposing my name here for the sake of wtf, don't pwn me.)

Two variables named somethingl and somethingI that had three grown developers scratching their head why the wrong variable was getting updated. This was back in the day when intellisense and colouring was way less advanced.

An actual compiler bug that wrote into the wrong register, went all the way down to assembler to realise the issue. This was on the early Xbox 360 (PowerPC) compiler.

Took a course during uni on computer graphics and one assignment I handed in needed correction, so I spent the better part of a summer trying to debug and find the flaw. Difficult to debug the graphics pipeline back then and with basically no tutors on hand during vacation times.

Turns there was nothing wrong with my thinking and program logic, I had just accidentally written two adjacent lines in the wrong order.

I transposed them, everything worked.

I was both relieved and pissed off at the same time. So clumsy.

I had a project I was working on a while back. I kept testing it and trying to find the bug that was causing my output to be completely wrong. It took me about 2 days to finally find it:

I had a boolean set to false, when it should have been set to true.

That one little bug completely broke my sorting algorithm. After changing it, it worked fine. :/

I was learning a new language, and decided to start with hello world. Only 7 words. I think I counted 14 bug fixes by the time I got it working.

I learnt (again) that I shouldn't program while tired, and really shouldn't learn a new language whole tired.

Used a niche language for specialised hardware that ignores the case values in switch statements.

Switch x { (x=0 executes this code) Case 123: (x=1 executes this code) Case 321: (x=2 executes this code) .... }

Also Java 7 timer object locks up when windows changes the system clock backwards during DST.

That I have a wetware problem.

When I'm debugging, I usually forget the nature of a bug when it is fixed. The more trivial it is, the quicker I seem to forget.

Something does trickle down to my unconscious because I would recognise getting the same error message over and over and that doesn't seem to happen often, but I could not tell you a single bug in the last thing I wrote, even though I know there were plenty of them.

What really makes this weird is that I sometimes become very upset - we're talking wailing and gnashing of teeth here - when I can't find a bug, and you'd think I'd remember those, yet once I do finally track it down and get over feeling angry and stupid, the nature of the bug still disappears from conscious memory.

Now, maybe I am learning lessons because of the infrequent recurrence of the same mistakes, but I'd like to be able to think about them later, and I can't do that.

midwest.social

Rules

  1. No porn.
  2. No bigotry, hate speech.
  3. No ads / spamming.
  4. No conspiracies / QAnon / antivaxx sentiment
  5. No zionists
  6. No fascists

Chat Room

Matrix chat room: https://matrix.to/#/#midwestsociallemmy:matrix.org

Communities

Communities from our friends:

Donations

LiberaPay link: https://liberapay.com/seahorse