That time that I worked at a programmatic click farm

From consulting to the sewers of the internet

At some point in my consulting years, I had reached I point I totally lost the motivation to move forward. Get assigned a project, do interviews, kick-offs, hunk down in a room for a bunch of hours and work out a deck with some obvious stuff but nicely presented and, after a round of multiple reviews, get it presented out to some director. Consulting firm and client goals and necessarily diverging (firm wants to bill as much as possible, client doesn't really know what too do and expect a deus ex machina to show up and fix everything) and the whole exercise slowly lost any reason to be in my mind and became more and more pointless. Consulting is soulless, uninspiring and requires to develop a good level of detachment from reality while having an extremely good grasp on a specific subject and being abrrrrrrle to seamlessly move from the most tactic to the most strategic level of thinking. You develop some amazing sales and problem solving skills which would definitely pay dividends in later stages of your life, but on a day-to-day basis can be pretty soul crushing. The competitiveness, long hours and constant pressure can be hard to bear. Sometimes I cringe when I think about the excitement and drive I had back in those days and I'm still amazed and how much I was able to accomplish with so little sleep. At some point all the excitement vanished and some weird type of anti-capitalistic pragmatism woke me up - this was all a huge waste of time, I had to make a change. I had made up my mind already when a friend reached out and talked me into joining a local startup. In all honesty, I had not really understood their business model at first - I was told it was a new affiliate network that developed their own programmatic attribution technology - but I quickly learned that the worst reason to join a company is to having to leave your current one as soon as possible. I wasn't really asking any questions anyway, I just wanted to leave and maybe spend a few months thinking about next steps. And that is usually easier with a paycheck, or so I thought.

What do companies really get when they buy programmatic media plans

I join an office with less than 10 people, one recent MBA graduate, 3 interns who would be forced to constantly renewing their internship to prevent them from be hired full time and two founders. Their pitch? In short, they helped advertisers generate leads and clicks by bundling ads with free software. You know the kind: those handy little tools like PDF converters or video players you download without thinking twice. At face value, seemed very legit and regular business. From a financial standpoint they were also doing reasonably well. The company was a spin off from a local media agency and it was constantly fed new clients eager to receive traffic and leads and run their promos. At that time, there were a handful of internet portals which were able to rank for the top spots for high volume keywords related to the most common software. Users would land on their pages first rather than go to the actual company developing the software to download the latest versions. The portal would monetize both traffic and downloads allowing third party to bundle their own installers with said software. At that time, there was very little transparency on what was actually getting bundled together. SEO was absolutely fundamental for these businesses - a few years later, a famous Google update removed a bunch of these sites almost completely from those SERPs. At some point in Mountain View, someone realized that you should not boost rankings of a third party software installer or at least prioritize them over the legit software source. One of this companies in particular, had to layoff 30% of its workforce due to the loss in traffic and the impact to their bottom line. Needless to say, they should have probably shut down altogether - they only contributed to make the internet a much worse place than what it really is. What was my role? The whole thing was so poorly glued together that they were struggling even issuing anything resembling a P&L sheet so I initially focused on the analytics and tracking side of the business, while pursuing a few different unrelated projects, as usually happens in small teams. But my scope was so poorly defined that I started struggling exactly what it was really needed versus what the founders really wanted. Needless to say I quickly understand this was not going to last for long.

The Installer: where things got murky

The company had a legitimate affiliate marketing business but they added their own spin to it, or so they though. Within a few days, I was introduced to the company's secret weapon: an installer program that partnered with popular free software developers. The idea was simple—ads were bundled with free software downloads, reaching millions of users daily and therefore creating a massive market waiting to be monetized. The installers weren’t just delivering free software. Hidden in the fine print of the install process was a checkbox—selected by default—that agreed to “third-party offers.” In reality, these offers weren’t harmless ads. They were paired with an “agent” that secretly installed alongside the software. That agent was effectively a keylogger. By this point, you should be already thinking of those annoying browser toolbar which were just so hard to remove. This was a step ahead of that. No visible toolbar, just a hidden agent downloaded unbeknownst to the user which would run in the background. Having a non-browser specific agent would have allowed for some more flexibility and not having to deal with any of the Microsoft or Google guidelines for developers. On the other hand, this also implied a different type of challenge - the agent was a legit executable running in the background, outside the scope of the browser. Now you might think that any commercial anti-virus or anti-malware software at that time was able to flag these installers. And in fact they were, although not as frequently as you might imagine. Out of 100 unique downloads, about 30 would never ping our servers, therefore we assumed they were flagged. This was a significant issue since if one installer gets flagged, their signature is shared across a network of security providers and multiple anti-virus software would have also identified it as a threat from that moment on. That would have been a critical threat, as the whole botnet would have potentially died overnight. As I remember, this was causing significant stress for the founders, as they started hiring foreign consultants which very interesting backgrounds in malware and security. We worked with a number of people from eastern Europe, likely legit black-hat operators, sometime trying to scramble a last minute update because we got word that some antivirus just flagged us. It was a constant cat-and-mouse game. The crazy thing I recall is that everybody in the office was so oblivious to anything it was happening - both because of lack of knowledge or really understanding the impact of what we were doing. All this while you could hear consultants in the background talking about code obfuscation, polymorphism, packing and all sorts of techniques trying to bypass the latest antivirus update. Download portals where also part of this since they run their own check on the installers being bundled. But guess what, they never raised a flag, since they never did their diligence. They were in fact a primary source for malware, although it took a few years for Google to truly make some changes and stop their SERP predominance. The funny thing  was also that the more they trying to hide and disguise the agent was generating more issues than the agent itself. Antivirus are primed also on these techniques, so they might even skip on recognizing the original pattern of the agent as malicious, but clearly pick up if someone is trying to hiding something in the code, no matter what it is. So what was a really strong business on paper, was hinged on some very precarious technical balance. The whole thing could have shut down overnight due to some large scale antivirus or malware update.

The truth about performance and programmatic advertising

As I started analyzing the ad performance, click-through rates were way above industry benchmarks. Clearly the agent were matching user search queries with relevant offers. While conversion rate were still extremely low, the sheer volume of traffic was enough that a minuscule conversion would have generated a significant upside anyway. Often, users clicked on these ads out of frustration or confusion, not genuine interest. And they would get another $0.10 in CPC. Then you get funneled into some weird form promising some rewards, and then bang, they made another $5 in CPL. And so it goes. Multiply that by a few millions click a day and you have a pretty nice cashflow with a barebones team and an adserver setup in some third world country. In a time in which Google CPCs were growing steadily, this was also a great way to build up some cheap traffic for most advertisers and beef up the media plans with some crazy discounts for media agencies. Thinking of this as a marketplace, if they had managed to find a buyer side with the installer and scale it pretty quickly, the seller side was also going growing steadily. They joined a number of affiliate networks happy to provide creatives for all sorts of CPL and CPC campaigns. With that sort of volume, the affiliate side of the business alone was incredibly profitable. But there I was, working for one of the most annoying things ever created on the internet, the heir of the infamous browser toolbar plugin. Someone did a good deal of research on this, and I can only recommend his book. At the end of the day, besides my own anecdotical experience and despite being a few good actors operating in the market, much of the inventory on programmatic exchanges is rubbish. Many publishers and marketers don't have the resources or knowhow to navigate this complicated landscape. As a result, enormous amounts of ad spend are wasted on inaccurate targeting, opaque pricing structures, and outright fraud.

Don't hate the player, hate the game (maybe both)

To this day I avoid to mention this job in my resume. It is funny that one of the worst experience I ever had from a professional standpoint, was also one of the most insightful and made me understand some of the inner workings of the modern commercial internet. Most importantly, it was a textbook example of gaming the system, and I couldn’t ignore it. Was it legal? I would say yes, but only because it wasn't explicitly illegal. Clearly this wasn't the first and won't be the last internet business based on some murky technical setup. However, every part involved was also completely aware of how deceitful and unethical this was. The download portals were funneling millions of downloads of unsafe software to uneducated users for the most part, affiliate networks never run any sort of diligence on the source of the traffic, neither as media agencies did and advertisers were mostly too stupid to even ask where the ads were run and the clicks were coming from. Latest I recall was that the company was likely shut down after a few months (probably it was only a matter of time before either the botnet was killed by some update or the investors truly realized how borderline illegal this was becoming). Companies like this can only last if they manage to find some legal avenue to grow and still make enough money to survive but they really unable to have any significant success - too much of a legal liability even in countries where this sort of things are not clearly regulated. One of the founders seemed to have rebuilt the same company in Latin-america, likely taking advantage of a more lax legal environment. In his Linkedin profile he boasts about bootstrapping multiple companies to million dollar exits, and he is know involved in some weird business based in some Mexican border town. One wonders what can take you from running a performance marketing business in Europe to a medical center in one of the most dangerous cities in Mexico... and becoming a CFO, as his profile mentions - a glaring proof of his ethical standing. He seems he also had the time to get a hair transplant - he looks much younger now. Funnily enough, my next job was at a media agency...

← Back to all posts