Cloud Transformation from the Trenches

I’m deep in the middle of managing my 2nd transition to AWS. Back in 2012, right before the first ReInvent I moved to put my startup's entire infrastructure on AWS. Back then Amazon had moved well past it’s S3 initial offering with key pieces VPC, EC2, EBS…but it was still early in the technology adoption cycle. And the story from Netflix was a confidence builder. My company, had hosted infrastructure in Rackspace that mostly scaled vertically and ordering hardware was painful. I knew the stack well and I could sense the opportunity in AWS to scale out, on demand, and most importantly use hosted pieces of infrastructure instead of building our own. In short, IT could now be a strategic opportunity to react faster instead of simply being a cost center. At the time I was new to management and I loved solving problems. After 10 years of software development, I’d finally learned that people are at the heart of solving tech problems. Leading a team was the best way to Get Shit Done with legacy systems. My biggest priority was how to cultivate motivation in my team, hire top talent, build complementary teams, and manage out cancerous attitudes. Now that I’m at my second IT transformation go-around some 5 years later, albeit at a much larger company, the playbook is becoming more clear. I’m dusting off the cobwebs of that experience and felt compelled to put my learnings on paper.

Transitioning to the cloud exposes practices and approaches your org should be doing, but isn’t. Do you have a CI/CD pipeline? Do you practice “build it/run it” mindset? Are you releasing often? How often do engineers have to tinker with production? How much system visibility do you have? How mature is your monitoring? Chances are if you haven’t moved to AWS yet, you don’t have great answers to many of those questions. Moving to the cloud and practicing devops go hand in hand because infrastructure is code in the new cloud model. Software is eating the world. Linux admins now need to work with VPCs, AMIs, Puppet, CloudFormation, Terraform, Docker, and Kubernetes. Software developers need to think about risk mitigation, operational posture, and monitoring. Committing to a migration plan means exposing things you should be doing, but aren’t. Moving to the cloud is an opportunity to leverage new technical capabilities, but to get there you need understand two things: what to stop doing so you can focus on building the future and where players fit on your team:leading the way, following along, or holding you back.

Prepare for automation
Go read The Phoenix Project if you haven’t already. While sophomoric and repetitive at times, the principles driven through narrative have a greater permanence than a dry and prescriptive checklist. I’ve worked with devops teams that have problems on opposite ends of the devops spectrum: from “throw it over the wall to ops” to developers running roughshod in production. Both types of teams can benefit from visualizing the work. What does the team do that isn’t written down? Do we have a SPOF (Single Point of Failure) on the team, like the infamous Brent from The Phoenix Project? How much of our time is spent on unplanned work? Is the team operating with an ’interrupt driven’ mindset? These are the questions you need to answer first. If your site goes down often, and the team is afraid to release software, you need to put on paper nebulous processes and system coddling. A good rule of thumb: anything that runs on production with some manual intervention needs a procedure. Now if you have a SPOF, this a great task to give them. These procedures become the requirements document needed for the long pole of automation.

As problems arise on the system, be sure to quickly run those to ground. The team must value that the site is available and reliable. This is the Give a Shit factor. If that’s true, then you can have effective retros and accountability. If that’s not true, you need to start thinking about organizational debt, and it’s time to manage out. Give a Shit factor is non-negotiable for any team, if you don’t have it you can’t create it.

Find the hidden work
I mentioned earlier making your work visible. My preferred method is Kanban to start. I don’t want to go into much more detail, as lots has been written about Agile, but suffice to say minimizing Work In Progress, daily standups, and timely retros go a long way. Getting company and team buyin can be the hardest part here, but once you spin this flywheel, you are on the happy path.

Finding hidden work is important because it exposes a value misalignment. If you have team members who are constant fire fighters, they have built their self worth on saving the day. Two values are important here: team work and kaizen. Firefighters solve for fixing issues, which makes it difficult to solve for long term system improvement. Work to find a person on the team whom you can build the team around and praise their diligence at digging out of the operational debt you may have. Other team members might have their pet projects. They’ve become disavowed with constant fires, and have resigned their sanity to a sandbox of unproductivity. The great thing about these people, is that you can give them a number of pet projects that can help solve for the long pole of automation or building the future.

Right people in the right roles
If you’re new to the team, it pays to listen first. As Steven Covey says, “seek first to understand, then be understood”. Don’t develop a prescription without sitting with your team during fires, listening to their thoughts on what they do well and what they don’t do well, and seeing how they normally get work done. So many times I’m tempted to jump in and start solving, but catch myself and practice patience. Likely there is quite a bit I don't understand early on. Maybe the devs are burnt out from too many fires. Maybe people have “we’ve always done it that way” attitude. Maybe you’ve got the brilliant jerk on the team. Maybe people don’t care. Those last two are deal breakers and are cancers on the team. You need to squash those immediately. Turning the narrative around can be hard and it can seem like you don’t know where to start. Slow down, listen to the team, have the hard conversations with individuals and teams. It will pay off in spades when the team starts running itself, freeing you up to look out ahead into future business strategies or getting to those side projects you’ve wanted to. The hard part comes when you realize that you are missing critical functions on your team or that you have team members that need to go. If you are missing a critical function, there might be creative ways you can reach outside of your team for help. Have a coach or consultant come in, if its coachable. You may also be able to tap resources on other teams, if you frame the problem to the business well.

Do more faster
Once you’ve cut your unplanned work down below 50% and once you’ve level set teams and individuals on some key issues, you can start diving into technical tactics. That is, you got some traction on your people and process problems, now you can focus on technology. In the last 6 years, devops has seen an explosion in productivity tooling. I come from the startup world where it’s all about ‘do more faster’. A key piece of this philosophy has been APMs. In general I hate operations, so the more I can focus on building or analysis the better. A good APM is to monitoring what the iPhone was to flip phones. New Relic, Dynatrace, and AppDynamics all have good offerings here. If you’re using Nagios or something similar, you are missing out on a tool that gives you system, database, JVM, CLR, webserver…metrics out of the box that also has basic alerting for when the system has breached baseline on any metric. The disk is full, memory spiked, database queries ground to a halt, and average http response times are spiking all works without a ton of custom tooling. Once you have your arms around how your system is behaving, you might need more granular custom tooling for subsystems. Sometimes you can build that custom tooling right into your APM. I’ve had great success at deploying simple metrics in a key code pipeline to visualize where the system spends most of the time and resources. For JVM and .Net you can get code level hotspots, that can unlock critical thorny bottlenecks in your system. Code hotspots along can justify the APM cost when you consider how much time your team has lost to troubleshooting enigmatic issues.

Bi-what?
So far these are basic dynamics have little do with the cloud. That’s not an accident. These are the key things that prepare your team for the transition, but here is the dark secret to this. I’ve had my best success taking teams toward a bimodal IT approach. One team is responsible for stabilizing and automating the old system, one team is responsible for the new go-forward paradigms. That is a contentious opinion. The team responsible for building the new system is less interrupt driven and able to keep steady progress so you don’t have to trade critical unplanned work for forward progress. The constraints are baked into the split teams. Now you can draw clear accountability. The key to making this transition is having the teams trust. They must trust that you’ll develop their career path and give them interesting problems. The biggest concern comes with the senior talent that runs the old system, that should be rewarded with new projects and greenfield. The reality is, that’s not where the complexity lies. The hard part is managing the old system, where the value lies, and creating space for the new system to develop. It’s not as if these 2 teams are in silos, there are key conversations happening back and forth, but again accountability is clear. The eventual goal is to bring the teams back together, or have some other team reshuffle. This split won’t last forever, but it’s typically the best thing for the business. Anything else, will take up too much time.

The other key benefit is maker vs manager time. In a heavy ops team that is responding to system issues, customer requests, and investigating bugs…it’s very difficult to get the headspace required to construct a new paradigm. I’ve read of teams that do a lift and shift, but that’s never been my approach. I’ve often leveraged new services and patterns that required writing software in conjunction with new AWS system architecture.

Moving to AWS is much more than changing infrastructure, it's a catalyst to look critically at the way you’ve always done things to make your company more competitive. In this regard I find more value in transformation than running operations. Beyond the transition to cloud and moving forward in operational evolution, and what I'm most passionate about is, using product and operational data to provide key insight into where the real business value is. Leveraging the organizational change from the AWS transition it's easier to build out a data strategy that will change the narrative from the bottom up. Perhaps another blog post.

An unconventional education

In the last two years I’ve spurned a proper career path for an independent route. In that time I’ve started a travel research project, took the GRE and considered going for a PhD, cofounded a startup, became a mentor for Defy Ventures, and spent 4 months developing a new ’entrepreneurial lab’ for a private equity investor. That last one ended abruptly and was a tough deal for me to move past. I had committed to jump in with both feet before the investor was fully aware of his machinations and the lab was shut down before it even started. I don’t prefer to dwell but suffice to say I learned a couple of key lessons. In my effort to shake off a number of false starts, I’ve felt the sense that can best be described as stuck. How do you move forward when you are stuck? “I think when you are truly stuck, when you have stood still in the same spot for too long, you throw a grenade in exactly the spot you were standing in, and jump, and pray. It is the momentum of last resort.” - Renata Adler, Speedboat

It’s been quite a ride and I’ve learned a great deal. Once you’ve heaped a great change upon yourself, that change can become an addiction. The personal growth derived from struggle is worth the agony. Yet, there is a fine line between trying new things and spinning your wheels. After a number of failed ventures, I find its time to move on from this phase of entrepreneurial efforts. That decision doesn’t come easy, it’s hard to leave after successive failures. But that’s just ego talking. I’m a firm believer in making lemonade from lemons and it pays to be creative. Before I move on, I’d like to put this period into perspective. I've long had a curiosity about entrepreneurship. For over a decade my sweet-spot has been helping a proven startup scale past the 1 - 2 million dollar mark. I felt the urge to put something into the world from scratch having spent so much time cleaning up the detritus of early startups. I can confidently say that I’ve given a sincere effort at building 0 - 1. I’ve studied books on entrepreneurship, consulted with entrepreneurs, and spent time building a business from scratch. I’ve learned more than an MBA could teach by learning AND doing in my time as an entrepreneur.

As it turns out, my transition period has opened up an opportunity to seize upon another life goal. For my parents generation the typical life phases looked something of the following: schooling (perhaps including college), work, family, then retirement. Today, attitudes are changing on the prevailing paths of contemporary professionals . The paradox is that your best and most capable years went to work, while your freedom to travel and pursue your passions was well after your physical prime in retirement. It’s a Faustian bargain to believe that the sacrifices you make now justify the for the rewards later. Momentum is a powerful force. This logical fallacy can contort your values over time. What you wanted 5 years ago may no longer relevant today. The pernicious hand of consumerism takes root and its easy to give our labors away for the lifestyle it affords. But that path has gnawed at me for the last 7 years. 13 years ago my father was diagnosed with terminal cancer, months after his retirement. My parents had bought an RV and put their home up for sale, ready to hit the open road. He had planned his exit for years. When I was a teenager I remember my father talking about the good life after he hits retirement. But instead of touring the country he made numerous visits to the Mayo Clinic for surgery and chemo, was forced to sell the RV, and plan for a more amenable lifestyle to terminal cancer. That has made an obvious impression on me. So painfully clear that we must take the time to do the things we want while we can. Why can’t we take the time to pursue other goals during our prime working years? After all, I’m fortunate enough to do so. Something I realize many people are not.

My wife and I have talked about taking a mini-retirement since we were only in the professional workforce for a couple of years. All my friends that know me, know that long term travel has been on my bucket list. But everyone, including myself, doubted that we'd actually make the leap. My wife and I are both competitive and career driven. Invariably over the last 15 years one of us has been ‘ready’ to hit the open road, but never both at the same time. Furthermore, as time goes on obligations grow and comfort settles in. Things like owning a mortgage, golden handcuffs at work, and wanting not for the recreational parts of our life have created barriers to exit. But the alternative is a life well lived. I’ve harbored the deep instilled narrative that it's worth forsaking these comforts.

As it turns out soon after my last failed adventure, my wife’s company was going through an Oracle acquisition. She has no interest in working for the the Oracle machine. Then the election happened. It’s time to get the fuck out of dodge. We’ve outlined a plan for a mini-retirement, and have taken steps to make it stick. Now I get to do things I’ve always wanted: learn a new language, live in another country, surf in exotic places, discover other cultures, and continue learning new skills. My philosophy on travel is that travel, unlike vacationing, is learning. Its active engagement and participation in the world at large. In fact, I’m extending my unconventional education. I’ve completed my studies in entrepreneurship and now am looking towards the humanities (anthropology, foreign language) and staying abreast of modern computer science trends (MOOCs in deep learning). The plan is as follows:

1. Take on an R&D role at my startup
2. My wife quits her job
3. Sell our house and store our stuff
4. Find a Spanish immersion program where I can surf
5. Plan an education in deep learning
6. Find other countries where I can surf and work
7. Budget for at least 6 months as a nomad

So far things are going to plan. I hired a remote dev to take over coding responsibilities at Brandcards. I was able to FSBO our house in North Boulder. An endeavor worthy of its own post. We’ve found a great immersion program in Bocas Del Toros, Panama. We’ll live with a local who will cook for us, attend 4 hours of Spanish a day, I’ll enjoy surfing in boardshorts, and my wife will enjoy the beach + yoga. My wife has just quit her job, and we have keyed in a few more places we might like to visit and practice our new language. I’m a big fan of simultaneous learning + doing. Our travels are built around the combination of learning and doing. I’ve targeted a MOOC in deep learning and books for my education in AI. I’m currently kicking the tires on a deep learning project. For now I've left out alot of details in favor of a simple overview. If you want follow along, we’ll be updating our adventures in our old travel blog . For this blog I plan to continue to explore software projects and perhaps how our travels can relate. For example, our Spanish Immersion program offers great parallels in AI. Stay tuned.

Systematic Innovation

When I first encountered the term systematic innovation it created a cognitive dissonance. The notion that innovation could be a process excited the deliberate thinker in me. However, I’ve worked in startups for over a decade and it seems like an oxymoron. Innovation is that bright spark or random event, not some derivative of Taylorism. Yet the possibility excites me, so I started doing some research.

First, it makes sense to explore the definition of innovation. The famous economist Joseph Schumpeter used the term 'creative destruction' he popularized in the 40’s. Clayton Christensen coined ‘disruption’ at the turn of the millennia. According to Schumpeter, the "gale of creative destruction" describes the "process of industrial mutation that incessantly revolutionizes the economic structure from within, incessantly destroying the old one, incessantly creating a new one”. A process which he thought to be driven by technology. Ken Norton, prodigious product leader at Google, thinks that innovation is a paradigm that requires a different way of thinking. Through attempting to improve a product or process by an order of magnitude are you forced to frame a different solution. 10x is a mindset, but a useful test for gauging whether an idea is innovative. Peter Drucker, the famous business and management guru, believed that innovation can be defined in terms of entrepreneurship. ‘The entrepreneur shifts economic resources out of an area of lower and into an area of higher productivity and greater yield’. That’s a dry interpretation and Drucker much preferred to avoid the theoretical, but he coined the term systematic innovation. He believed that there was a methodology by which opportunities could be found and exploited:

Inside an Industry:

• The unexpected - the unexpected success or the unexpected failure
• The incongruity - between reality as it is or reality as it's assumed to be
• Innovation based on process need
• Changes in industry or market - that catches everyone unaware

Outside the Industry

• Demographics
• Changes in mood, meaning, and perception
• New Knowledge, both scientific and unscientific

If Peter Drucker was extremely conservative in risk taking, Mervin Kelly, former head of Bell Labs, was the opposite. R&D is a spectrum from pure research, to applied research, to exploratory development, to early stage development. Mervin Kelly relied on more fundamental forms of research that could then be piped into development. “Inventing the future wasn’t just a matter of inventing things for the future; it also entailed inventing ways to invent those things”. He believed that R&D could be a formula and worked to create the proper conditions for innovation:

• the world’s best scientific minds
• thinkers and doers under one roof “a physicist on the way to the cafeteria was like a magnet rolling past iron filings"
• trusted people to create and encouraged idle curiosity
• record everything in notebooks
• a problem rich environment

Many methodologies have ties to each other. For example, Bell Labs relied on good problems. Lean Startup begins with the problem definition. Clayton Christiansen relies on the ‘jobs to be done’ mindset and Kelly, thought innovation was a job to be done ‘better, cheaper, or both’. 10x thinking is present in many contemporary writings on startups. It’s where the commonality exists, I find value. Great thinkers build on those of the past and the right way of framing a problem has permanence, but is utterly simple. When Charlie Munger was asked why others haven’t followed his methodology, he quipped “because it’s too simple".

My research into systematic innovation has left me with practices of my own. When I read, I take notes. I choose what to read carefully. I write down what Drucker would consider key opportunities to exploit. I’m aware that innovation is a balance of passion and methodology. Passion can create a blinding bias I commonly refer to as ‘founder syndrome’. Methodology will keep you on the right track, but won’t keep you persisting if you don’t have passion. I look for good problems, not good ideas. I use the 10x filter. I believe that entrepreneurship is a tool. When you think you have a good idea, there are practices that can bring that notion to market effectively. Innovation requires with a proper framing of the problem, but it takes a keen mind to spot opportunities.

An Original Work

For the last couple of years I’ve been curious on the nature of original work, or invention, in so far as software is concerned. I’ve had the fortune of speaking with inventors, patent holders, fresh PhD graduates, and entrepreneurs. Each time I meet these people, I feel engaged. Engagement is that notion of true interest and focus that I’ve lost perspective on over the years. Work turns into momentum and somewhere along the way I lost my engagement. I can't remember meeting someone who holds a PhD that wasn’t engaging. Perhaps for 2 reasons: first at some point they opted out of the comfortable, but traditional path of private industry and, second, they’ve spent considerable time devoted to more graduated and critical thinking. More than once I’ve considered going back to school. I even took the GRE last fall. Ultimately, I couldn’t settle on a field of study that I was excited enough about to devote a period of my life to. My favorite summary of why to get a PhD: "Pursuing a Ph.D. is the only way to spend 4 to 8 years being paid to work on something that the market does not directly value in the short term." I haven't found anything yet that I'd like to make that long of a bet on, but there are a few questions that could be good canidates: how can machines interface directly with the physical brain to expand cognition, can machines ask good questions, can nuclear energy be safe and affordable. I believe all of these to be in the realm of possible...it could be now or in 50 years. I’ve had some conversations with PhD students, over many beers, who were disillusioned with the process of choosing a focus for their studies. My limited impression is that you apply to school thinking you’d love to study X. But then you realize that there are only Y professors taking on new candidates, even fewer with whom you have a chemistry, and even fewer who have funding to support your work. A lot of work to not be doing what you’d like to be doing. Additionally, you have to enjoy the process, and having been a pragmatist most of my working life, the idea of research is much different than engineering/development: your work is not a means to a specific end.

For now I’ve put my academic ambitions on hold. I've turned my attention towards inventors and/or entrepreneurs. I believe there are 2 broad notions of producing an original work: academic or hacking. I’ve been a hack for sometime. Make it work, make it right, make it fast. The lack of discipline in that approach can have it’s own pernicious effects. In fact, if you watch this fantastic talk on the background behind the recent resurgence in neural networks, you can draw a line to Geoffery Hinton who can draw a line to many other researchers going back to the 70’s. Geoffery makes a strong argument for the case of pure research. But the very nature of how he came up on this discovery proves that his research could easily have not produced something so applicable outside academia. For decades neural networks were a mostly academic exercise until he found a couple of key revelations. The previously mentioned article on research vs tinkering strikes a chord and makes me want to be more disciplined in my hackery. Not to long ago I did some of my own hackery on LDA as it relates to travel blogs. Likely, somebody has already done similar work, but I did little research to build on previous work. I also was quite sloppy in execution. However, my aim was to address a basic premise: was this problem solvable? To that end, I enjoyed the process very much. A personal challenge of sorts, similar to the reason do long distance triathlon, speak publicly, or venture into surf outside my comfort zone: I aim to push myself and grow.

There are innumerable ways for me to produce an original work. I've decided to focus on 3 tenants: the work should be applicable at some point in the near future (less than 2 years), that work must be distinct enough from other works which can include how that work was executed, and, finally, I’d prefer to leverage my existing talents that exploit knowledge across unique domains. My current expression of that original work is through a new startup that aims to disintermediate traditional retailers and advertising by connecting brands more directly with consumers. If you are one of the 2 followers of my blog, you'll note this is a departure from my consulting adventures. With good reason, I need to scratch an itch to create something from nothing. There are many interesting machine learning challenges to tackle in this new startup adventure: recommendation systems, identifying unique products and their variations, and reinforcement learning. Within that space, I aim to do some exciting work that will, no doubt, leverage lots of other’s work. Perhaps I might be able to leverage the work of an academic whose efforts haven't yet seen production. A marriage of academic and entrepreneurial endeavors seems a fitting compromise.

Investor CTO

For the past 6 months I've contributed my time to 2 startups. In that time, I've become keenly aware of two things: 1) there are a lot of startups that need technical and management help and 2) my background in software development and executive leadership enables me to provide valuable advice and hands-on implementation to help those startups. The model is simple: I take a reduced compensation rate in addition to an equity stake in the company that has an attached pre-defined equity maturation event. Now my time is limited, so 2 or 3 startups constitutes a full book at the moment. It pays to be selective like most investors are.

I've taken alot of meetings with founders and would-be founders. Building a pipeline of potential clients is as much work for me as it is the startups I work for. I see founders that fit in 2 camps: those that have deep expertise in a domain and decide to take a run at making a business out of it. We'll call them domain founders. Then there are those that have deep experience working for companies and are trying to make a go at a market that is new to them. We'll call them experienced founders. With domain founders, it's important to be able to work with someone who knows that they can't do it all and is open to "coaching up". I tend to provide input well beyond technical aspects and more into how to scale or growth-hack various parts of the business. Experienced founders typically underestimate how the lack of knowledge in a new market will hinder their path. I call them experienced founders because they know how to scale at a certain point, but they lack that zero to one growth experience. With these founders its key to emphasize how risk is viewed differently in a startup. Getting to market fast, cutting scalability corners, honing in on your hypothesis, and testing the market in various ways is key.

My most selective criteria is whether I can work with the team. I'll take an initial meeting and a first engagement at no cost. It's worth it to invest my time upfront for free if there is an opportunity. What makes a good team? Proper role definition and open mindedness go a long way in my book. They say you need a hacker and a hustler. That's a good attempt at initial role definition. Often times, I'll come into a company that doesn't fit that mold, and you have people stepping on each others toes. That's fine, but it needs to be sorted quickly. And that process is never ending. Being open minded, is a must-have. Nobody can predict the future, as Daniel Kahneman states "experts are no better at predicting the future than dart throwing monkeys". If there is contention, state how you intend to objectively move through that disagreement and what the risk is associated with you being wrong. I've been hugely wrong. But in the end, I was glad to be proven wrong, because that was a big win the for business. Third, I'm a fan of the no asshole rule. I've met founders that are bullies and run over a conversation. They interrupt their people and generally think they are God's gift to free market enterprise. You spend more time working with people than you do your family, so my investment is more than fiscal. I need to enjoy working with these people, and assholes just won't cut it. Finally, I need to work with competent people. Both intellectually and emotionally competent people. There is a famous TED talk that says "people don't buy what you do, but why you do it". Often times that 'why' can get distorted and convoluted . I would restate "people buy who you do it with first over what you do and why you do it". Team is number 1 in my book. And it seems that I'm not the only one. In a recent survey of software devs, culture and colleagues rank above building something significant.

Second to the team, is knowing how my skills can help this business succeed. If you're working with hardware, I'm not a great fit. If you're working with enterprise software, I'll pass. If you need to build an MVP, I'm interested. If you need help managing your development team, I'm all ears. If you're trying to raise capital, I can help you scope your efforts and be an experienced asset on the team. It pays to tell people 'no' and be specific about my value-add. After all, I'm expecting these fledgling companies to give me a longer term stake in their business and I have to feel that I can provide value.

As the team and product grow and scale, I get to see who isn't pulling their weight. I don't time for passive equity holders. A new startup will go through phases. Maybe there are more technical needs in the beginning, and marketing needs to hold off. This is why role definition (which changes over time too) is important. Accountability needs to be clear. Startups go through phases: building an MVP, getting early adopter traction, scaling past $1 million, raising capital... I like to have a good sense for who is going to contribute and how much. If I see passive equity holders along the way, I will voice my concerns. After all this is an investment for me, and I don't need dead weight pulling down my investment. Warren Buffet's model is simple "estimate an investment’s intrinsic value, handicap its risk, buy using margin of safety, concentrate, stay in the circle of competence, let it roll as compounding did the work.” I believe as a partial investor in a company I can provide much more value for the long term and decrease the odds of failure substantially.

One aspect of this position that I enjoy more than I ever thought is the creative thinking you can apply to building a business. Tech is full of business models that never existed. In fact, the concept of "business model" didn't really exist until the dot-com tech bubble. Adjusting to the constantly changing field of information: competitors, market, capital...is full of opportunities to exploit. If you can't see the opportunity in things, startups are the wrong game for you. As John Maynard Keynes states "A large proportion of our positive activities depend on spontaneous optimism rather than on a mathematical expectation”. I enjoy the possibilities of the game. And that is likely the biggest draw for me. I could make more money in a corner office, but I much prefer the wild west of startups. As Warren Buffet states "we enjoy the process far more than the proceeds".

Being an investor-CTO is as much a new adventure for me as the startups are for those I consult for. I still have alot to learn. But Boulder is a burgeoning town for startups. And I'll never close the door on joining one of these teams fulltime.

Short term travel for the vagabonder wannabe

Traveling is best when there is an open ended expanse of time. But what if that's what you don't have? For the last decade, my wife and I have longed to head out on that open-ended wanderlusting journey, but career ambitions have gotten the better of us. Thus, we’ve been crafting the art of short term ( 1-3 weeks) international travel. What follows is an outline of the mindset and approach needed to make the most of your limited time. Before you read on, I assume that you:

• like to experience a trip instead of ‘checking-out’
• you’ll have more to spend (particularly on logistics) than the long-term vagabonder
• you see the value in a flexible itinerary
• you’d like to avoid the traditional tourist line when you can

If that’s not your cup of tea, you may find the suggestions in here not fitting to you. If you typically book cruises, close this browser immediately.

Research

First and foremost, be researched, but not regimented. There are young travelers and newbs who need to have an itinerary, but a flexible plan is better. What if the weather sucks? What if you want to stay longer or shorter? How do you stay flexible? It’s so much easier now that’s it used to be. Our trip to Italy, 10 years ago, we’d start calling down the list of Lonely Planet’s accommodation section the day before, but more often the morning of, our intended arrival. Now, Airbnb, cheap international data plans, and a few handy smartphone apps make flexible travel much easier.

What does it mean to be researched? Logistics, areas of interest, and knowing what needs to be planned. Let's start with areas of interest. I'm assuming you know where you want to go. But let’s be more specific. What is there to do in a region? What are the highlights? Food, adventure, sights, culture... Use TripAdvisor and Google to get a lay of the land. Read travel blogs to find compelling experiences. You are looking for depth here not the postcard stuff, but what's underneath. Time of year, touristy rating, cost, time commitment... That's the depth you are looking for. Be judicious in determining whether your content was paid by an interest other than traveling (like a hotel chain sponsoring a blogger, a country promoted post). Not that sponsored content is useless, but the intentions of the author can be misleading and often outright deceptive. This scathing review of a TripAdvisor post is just the tip of the iceberg.

Logistics

Now if you have strung together a few places of interest, start flushing out logistics. How do people get around: train, bus, car, motorhome, walking, scooter, bike...Travel forums and travel blogs are great at digging this stuff out. Renting a car buys you the ultimate freedom, but knowing the bus/ferry/train schedules can work well too. Don't be intimidated, but read up on other's impressions of those transportation modes respective to your frame of reference. For instance, do you drive on the other side of the road? Do trains tend to run late? Is English common? For instance, in Croatia, the ferry schedule changes with the season. Finding accurate info online can be a challenge. This is why I created a search engine based on narrative travel blogs.

Commit at the last responsible moment

What needs to be planned and what can wait? Do you need to book that tour in advance because it sells out? Obviously you need plane tickets there and back. What about inter-region flights? Only a few places to stay at a particular area? Book it. The more you can widdle this down the better. Here is the key: there are lots of things to do in a region and reserving your commitments to the last possible instant increases your chance of finding that key piece of information that might totally change your mind. For instance, we did a zip line tour that we booked hours earlier. Most people book, weeks or months in advance. For us, it turns out the tour was on the way between cities we were traveling to. Lots of people recommended we visit Plitvice Lakes in Croatia because of the expansive beauty and unique landscapes. Well the weather was crap when we would pass by this area, and another less busy park, was convenient for our itinerary and the weather was great. You could also go swimming there and its was less crowded. We decided to go there the morning of. It’s these last minute plans that seem to have the best effect on perceived travel experience. Serendipity, can bring a lot of joy to a trip. Working to find serendipity is a bit like hammering water to make it flow, but you can create the conditions that improve the chance of making it happen.

Sample Itineraries

You need to put it all together by knowing where to start. What direction to head? Sample itineraries? What-if scenarios on that itinerary. There are some sample itinerary websites out there. You can get a sense from ’trip blogs’ of where to head. Q&A travel sites can also help with itinerary generation. I posted this on Trippy and got 3 pretty decent answers. In the end, we didn't pick any of these specifically, but the information provided was helpful.

On-the-ground planning

Lastly, I relish the experience of piecing things together on the ground. You can pick up information from locals or other travelers. Once you are on the ground use Google Maps, TripAdvisor, saved information from your travel investigations, travel books, and importantly Airbnb. Lots of times the host will provide you some key information in making the most of you time. On the ground intel, adjusting, re-routing, last-minute decisions...helps to seed those moments of serendipity.

Topic Analysis In Depth - Croatia Travel Blogs

I'm headed to Croatia soon, so I decided to try another run at topic analysis from more than a cursory look. One of the challenges I've had with various topic analyses was validating the result set. Recently I've discovered LDAVis which does just that for a topic analysis. I won't go into LDA too much here, but Ted Underwood does *the* best job at explaining LDA in an intuitive way. LDAVis provides the means to discover conditions of your topic through the visual relation to other topics, how distinct topics are from one another, and term frequency/saliency within topics. Here I'll walk through how I used the tool to execute many different topic extraction/data cleaning runs to reveal interesting insights into Croatian travel.

The dataset includes blogs scrapped from travelblog.org and travelpod.com where I included information from the body and title. I have this dataset locally on my computer in ElasticSearch, which includes 1500 blogs on Croatia alone. ElasticSearch isn't really necessary, for the analysis itself, but it was convenient for document storage, term discovery, and blog review. Below is a sample screenshot from my "Croatian" version using a custom search tool. You can see that I searched "organ" which showed up in one of the major topics, and that struck me as odd. It turns out that there is a pipe organ in the city of Zadar called Sea Organ. (described here, 2nd paragraph from the bottom). It plays music by way of sea waves underneath a large set of marble steps.

If you looked at that last blog, that is a typical representation of a good travel blog: descriptive, narrative, lots of pictures, and focused on a distinct region. It's important to note that these blogs are mostly journal-style catalogs, not 'Top 10' posts. The compelling notion is that, theoretically, whatever you find in this data are real experiences that people have had. As you can tell, I'm a fan of this dataset, but it can be pretty messy to analyze. Some things I've had to account for:

• difficult to filter HTML (even using BeautifulSoup for the main post of the body)
• only filtering english blogs (using a package called LangId)
• removing stop words ('a', 'the', 'an'....). I used a much more expansive list than the out of the box NLTK one.
• removed people names. more on this later.
• remove words that appear infrequently

Before diving into the topic analysis, a great way to discover some features (and bad data) is by looking at some different n-grams. Basically, what are the most common single words, word pairs, and words that appear in 3's.

Popular Unigrams (starting with most popular)

• day
• town
• time
• city
• croatia
• water
• dubrovnik
• people
• split
• good
• bus
• night
• boat
• island
• walk
• beautiful
• well
• small
• great
• walked
• place
• dinner
• will
• bit
• park
• sea
• nice
• trip
• beach
• decided
• going
• morning
• tour
• croatian
• lunch
• zagreb
• today
• hotel
• road
• headed
• didn
• find
• left
• long
• hours
• restaurant
• walls
• arrived
• area
• hour
• car
• local
• couple
• top
• hvar
• pretty
• side
• days
• wine
• walking
• early
• lovely
• apartment
• coast
• lot
• ferry
• room
• lakes
• food
• breakfast
• head
• big
• ve
• view
• palace
• amazing
• group
• ride
• pm
• streets
• sun
• don
• lots
• minutes
• bar
• best
• started
• stopped
• main
• set
• called
• full
• large
• views
• finally
• train
• adriatic
• tomorrow
• told
• swim

Popular Bigrams

• national park
• city walls
• ice cream
• plitvice lakes
• adriatic sea
• diocletian palace
• bus station
• cable car
• game thrones
• walking tour
• walked town
• walk town
• stari grad
• years ago
• early morning
• crystal clear
• couple hours
• tour guide
• walled city
• cruise ships
• bell tower
• half hour
• train station
• hvar island
• bus ride
• marco polo
• cruise ship
• long time
• small town
• hvar town
• main square
• day dubrovnik
• island hvar
• full day
• narrow streets
• decided head
• pretty good
• town dubrovnik
• dalmatian coast
• minute walk
• olive oil
• plitvice national
• split croatia
• unesco heritage
• lakes national
• rest day
• bosnia herzegovina
• day split
• day trip
• decided walk
• heritage site
• boat ride
• city dubrovnik
• walk city
• upper town
• white wine
• arrived dubrovnik
• broken relationships
• caught bus
• dubrovnik croatia
• great day
• spent time
• early night
• long day
• top hill
• well worth
• headed town
• main street
• museum broken
• people watching
• rental car
• blue water
• couple days
• grocery store
• red wine
• roman emperor
• tour group
• beautiful city
• good time
• shops restaurants
• top deck
• air conditioning
• clear water
• sail croatia
• side road
• blue cave
• day day
• emperor diocletian
• views city
• amazing views
• beautiful place
• glass wine
• great time
• pile gate
• upper lakes
• bus town
• day croatia
• island brac
• lokrum island
• side island

Popular Trigrams

• lakes national park
• plitvice national park
• plitvice lakes national
• museum broken relationships
• unesco heritage site
• krka national park
• roman emperor diocletian
• crystal clear water
• walk city walls
• clear blue water
• water crystal clear
• game thrones filmed
• marco polo born
• walking tour town
• bad blue boys
• cable car top
• main bus station
• second largest city
• zagreb capital croatia
• hour bus ride
• largest city croatia
• caught local bus
• good nights sleep
• st mark church
• crystal clear waters
• hula hula bar
• national park croatia
• town stari grad
• walked city walls
• white water rafting
• austro hungarian empire
• built roman emperor
• cruise ship passengers
• decided head apartment
• eating ice cream
• ice cream shop
• mljet national park
• national park plitvice
• spent couple hours
• town hvar island

Here is the simple source code

def ngrams ():

    corpus = ''
    for blogtext in getblogs ():
        corpus += blogtext

    stops = longstopwords()

    prime_words = [word for word in utils.simple_preprocess(corpus) if word.lower() not in stops]

    bigram_measures = nltk.collocations.BigramAssocMeasures()
    trigram_measures = nltk.collocations.TrigramAssocMeasures()

    finder = BigramCollocationFinder.from_words(prime_words)
    finderII = TrigramCollocationFinder.from_words(prime_words)

    # only bigrams that appear 3+ times
    finder.apply_freq_filter(3)
    finderII.apply_freq_filter(3)

    print "TOP UNIGRAMS:"
    for t in nltk.FreqDist(prime_words).most_common(100):
        print '<li>' + t[0] + '</li>'

    print '-----------------'

    print "TOP BIGRAMS:"
    for bigram in finder.score_ngrams(bigram_measures.raw_freq)[:100]:
        print '<li>' + " ".join(bigram[0]) + '</li>'

    print '-----------------'

    print "TOP TRIGAMS:"
    for trigram in finderII.score_ngrams(trigram_measures.raw_freq)[:40]:
        print '<li>' + " ".join(trigram[0]) + '</li>'

You can see from the code that n-grams can be a simple and powerful way to do some discovery on your dataset. If you take a look you can see 'Game Thrones' in the Bigrams list. 'Of' was filtered out in stop words, but a good deal of filming for the Game of Thrones takes place in Croatia. Numerous national parks and towns are in these lists as well. Searching these terms into my search engine yields some interesting discoveries and I've already discovered an activity of interest: white water rafting.

Below is a sample of LDAVis UI for topic analysis. I debated whether I should write this post in IPython (where I did the LDAVis analysis). Ultimately, I like having my posts in a single place. But IPython (now Jupyter)is a convenient way for analysts to share their work. You can now run these notebooks off of GitHub , and I stuck the best of my data runs and data in this Jupyter Notebook. The process of getting at this result, what this interactive chart says, and the process of getting there are worth some note. As you can see there isn't as much distinction between the topics as I would like, as noted by the overlap in topic circles. However, the results still provide a good bit of insight. Topic #2 contains 'zadar, water, beach, sea, organ' as some of the most frequent+salient terms. Basically, the word 'organ' doesn't appear much in the corpus but when it does it belongs exclusively to this topic. When the lambda slider is set to .5, we see a good mix of popular words vs strong words. Lambda is essentially a balance of frequency and saliency that you can control. Some topics were awash in terms in ambiguous terms, but manipulating Lamba can provide clarity. Topic #2 is what gave me pause to investigate "organ" and discover the Sea Organ in Zadar.

I used Gensim again to perform the LDA analysis. This library takes lengths to scale and be performant, which is important so I don't have to switch to a new library to perform large analyses. Here is the sample code used to produce my corpus, dictionary, and LDA. The general process is as follows:

1. Get blog text from ElasticSearch
2. Get an expansive list of firstnames and stopwords to filter out
3. Only include terms that appear 4 or more times
4. Generate Dictionary, Corpus, and LDA model

def Dictionary_Corpus_LDA_Withoutnames():
    
    #get array of blogs stored in ElasticSearch
    documents = getblogs ()

    #get a list of first names to filter out (Gathered from US Census + half a dozen foreign names + nicknames)
    firsts = getfirstnames ()

    #get a long list of stopwords to filter out
    stops = longstopwords ()
   
    words_to_exclude = stops + firsts.keys()

    logger.info("Removing stop words and names")

    documents = [[word for word in utils.simple_preprocess (doc) if word not in words_to_exclude]
              for doc in documents]

    logger.info("Removing words that only occur once")

    token_frequency = defaultdict(int)
    for doc in documents:
        for token in doc:
            token_frequency[token] += 1

    #remove words that occur 3 or fewer times
    documents = [ [token for token in doc if token_frequency[token] > 3]
                for doc in documents  ]

    logger.info("Saving Corpus and Dictionary")

    dictionary = corpora.Dictionary(documents)
    dictionary.save('CroatiaTravelBlogs_NoNames.dict')
    corpus = [dictionary.doc2bow(doc) for doc in documents]
    corpora.MmCorpus.serialize('CroatiaTravelBlogs_NoNames.mm', corpus)

    topics = 50
    passes = 100
    lda = models.LdaModel (corpus, id2word=dictionary, num_topics=topics, passes=passes)
    lda.print_topics(topics)
    lda.save("Croatia_Topic_No_Names-{0}_Passes-{1}.lda".format(topics, passes))

Take a look at a few more topics listed in the notebook. Set the Lamba slider on the right graph to .5 and choose a topic number on the left graph. For example Topic #4 is a bit confusing. Toggle the Lambda down to .33 and you'll get 'bus', 'hostel', 'squirt' (nickname I should have filtered)..'cheaper' and 'backpacks'. I think this topic contains blog for budget travelers. Tagging these blogs for would-be travelers could be very compelling feature to the search engine. I'm unclear on Topic #1 and #3. Topic #5 is about Dubrovnik. Topic #6 is relates to the popular islands just off the coast. Topic #7 is about the history and architecture. Topic #8 is about camping and motorhomes. Topic #9 is about some of the national parks. I think you get the idea. To reiterate, I often move back and forth from the LDAVis to my search to check out terms in context of the blog. That's how I knew that I missed a nickname in Topic #4 since it's always the same author and the first few results were about their son.

I didn't go too indepth on LDA here, but its worth reading the blog by Ted Underwood mentioned above. The intro to LDA written by one of the original authors, David Blei, is here. The paper behind LDAVis is interesting too. he website is best to get a better intuition for their methodology of LDA analysis. Finally, if I were to take the exercise further academically, I'd improve LDA specifically for blogs by adding another dimension for author and custom treatment for title.