May 22, 2015

On Quitting as CTO, Wantrepreneurship, and Starting from Scratch


Shitting my pants before bungee jumping
Two weeks ago I quit my job as CTO.  With long aspirations to become CTO, it was a surprise to many, and at the time I realized it, to me as well.  As CTO you get to help shape the company in a significant way, which can be very rewarding.  From company to company the role can vary greatly as Werner points out.  But the reality is at the executive level you have to deal with an extraordinary amount of bullshit.  It's just part of the job.  As the company grows, leadership needs to have consistent values, messaging, and scalable operations.  Reticence and self discipline go a long way at a company that has its fair share of organizational and technical debt.  In fact, I've become convinced that what everybody in HR already knows: emotional intelligence is truly a defining characteristic of most good executives.  I was fortunate enough to have a sense of this from before I even started the workforce.  When I was a teen in the early and mid 90's the nature of intelligence was in debate and I was soaking it all in.  A book called Multiple Intelligences came out then a highly controversial book The Bell Curve and then Emotional Intelligence after that.  My general takeaway after this debate was that intelligence is simply the ability to adapt to one's environment.  That's a hotly contested opinion, but one that's stuck with me for a long time.  But here's the catch, you have to pick the right environment.  After scaling with the company for five and a half years, I learned that servant leadership also means that you might work yourself out of a job that you actually like.  Having considered myself a change agent for so long I realized I no longer had the drive to push things forward.  It was time to step out of the way to let the talented team I'd worked so hard to build, step up.

Being as it was, the decision didn't come easy.  My disposition is typically all-in.  I knew once that I'd lost the desire for my current job that I'd feel like a fraud.  It was important to me that I wasn't one of those employees phoning it in.  I absolutely hate that.  I'd sooner do something that's whole ass than half ass.  Second, the timing for me financially is amenable to being without income for a period.  I've deliberately lived my life free of financial burden and undue American materialism.  I have a reasonable mortgage, no other debt, no kids, and a wife that has a more than adequate income to sustain our standard of living.  The toughest part of quitting was leaving the team I'd hired.  Who you work with is paramount in my book and this statement from Peter Senge in the Learning Organization really strikes a chord with me
"When you ask people about what it is like being part of a great team, what is most striking is the meaningfulness of the experience. People talk about being part of something larger than themselves, of being connected, of being generative. It become quite clear that, for many, their experiences as part of truly great teams stand out as singular periods of life lived to the fullest. Some spend the rest of their lives looking for ways to recapture that spirit."
I've been part of a great team and when I was able to build my own I sought to define and establish a set of principles needed to rebuild something similar.
  1. Create growth opportunity and narrative 
  2. Find those who show an aptitude for continual learning 
  3. Find those who work well with others 
  4. Make sure the role and growth opportunity is defined 
  5. Only hire those who give a shit
The good news is that with a dugout full of talented people they'll lean on each other and their own talents to find their path forward.  Of that much I am certain.

Those that know me well, know that I've had intentions on starting my own business for a long time.  Being an analytical person, I wanted to better understand the landscape.  By working for 4 startups in 10 years, I got to learn a lot.  As a developer I learned how to build and maintain software.  As a manager I exercised my people skills.  As an executive I've exercised my business acumen. Brad Feld talks about Wise's Talent Triangle in Startup Opportunities and I have 2 of the 3 covered: business acumen and operational experience.  So in essence, I think I have enough of a foundation to find success in the entrepreneurial game.  But unlike the rest of Brad's book I don't really meet the checklist of knowing when to quit your day job.  I'd give myself a 7/10.  But that's passing, right?

I'm sure there are many stories of how folks decided to quit their jobs.  It's not like I have this project on the side that's gaining traction and it's not like I have a co-founder who has wanted to start something.  Generally speaking, I'm at a pretty open stage with nary a plan.  I do have an idea that's nagged at me for sometime.  It's in a space I'm passionate about, but have little experience in.  The technology approach is appealing to the nerd in me, but I'm still trying to discover the market opportunity.  At the very least, I'd end up with some compelling technology discovery and perhaps a way with which to leverage that into a product will become clear.  I need time.  Time to explore and create.   It's been too long since I've created things or worked on a team that's building something.  I'm most eager to get back into that seat, get more hands on with coding and data science.  That in and of itself will be a win.

So far, it's an emotional roller coaster.  Having been at the company for so long, part of my identity was wrapped up in it.  Combine that with the fact that I'm not racing this year, I'm a ship without a rudder.  But it's also a good opportunity for reinvention.  I went to Boulder Startup Week, and got some great advice from a few mentors.  I've also reached out to old colleagues and old mentors.  It's really encouraging to have a good support network in my community and from my past in general.  But in the end you need to distill all the information you get into a direction.  Lean Startup is all the craze these days, but it's no silver bullet.  You can't substitute methodology for good ole fashion critical thinking.  Time to get to work.

Mar 7, 2010

Nullable method parameters with FluorineFx

After upgrading to the latest version of FluorineFx, we noticed quite a few new exceptions: "Could not find a suitable method with name %". We checked the parameters, overloads, ect.  One thing was consistent, each had at least one nullable parameter.  We've already branched FluorineFx for datetime issues (TimezoneCompensation.None doesn't actually mean none, that's another post), so I took a crack at fixing this one as well.  I traced everything back to the bloated method TypeHelper.IsAssignable().  As best I can tell this tries to see if the method parameter in-hand can be assigned to the parameter type of the method.  At the heart of things, its using the .Net TypeConverter, but it won't handle nullables.  You need to use the NullableTypeConverter instead.  We added the following line of code to line 693 of the TypeHelper file:

........
if (obj != null)
{
    if (isNullable)
    {
        NullableConverter nullableConverter = new NullableConverter(targetType);
        targetType = nullableConverter.UnderlyingType;
    }

    TypeConverter typeConverter = ReflectionUtils.GetTypeConverter(obj);//TypeDescriptor.GetConverter(obj);
........

Why don't I just check this into trunk?  Well I tried to contact Zoltan, the main contributor to FluorineFx as far as I can tell, and he's completely unresponsive.  Bummer.  There a few other nitpics we'd really like to checkin.

Feb 20, 2010

Custom Error Reporting with log4net

I recently started a new position where hunting for errors included logging into one of two active web servers, looking over a couple of directories that were logging via log4net, and also checking the Windows event log.  Needless to say this was a PITA.  I decided my first initiative was to try and improve the visibility into our application errors, to better understand our production issues.  To confound the issue we weren't getting context like server variables (browser, referring url, ect) or the user logged in, which can be very helpful in the discovery process and also for support.  Typically I would try to use something like Elmah, because the less work the better, but there are a few snags.  One, we are using a custom db session provider which helped to link the dying ASP pages to .NET.  Two, we use Fluorine and NHibernate, and they do alot of internal logging using log4net.  Additionally our existing app had log4net logging all over the place. So I decided to set out on a custom appender to consolidate.  There were a few configurations I thought of, but I settled on inserting all errors into the database and using an admin interface to view, datamine, and manage our exceptions.  First thing I had to do was insert a Global.asax in all 8 of our applications to catch all unmanaged exceptions. Each one had something like the following:

void Application_Error(object sender, EventArgs e) 
{ 
    // Code that runs when an unhandled error occurs 
    log4net.ILog log = log4net.LogManager.GetLogger("MyApp");
    if (log.IsErrorEnabled)
        log.Error("An uncaught exception occurred", this.Server.GetLastError());

}

void Application_Start(object sender, EventArgs e) 
{
    // Code that runs on application startup 
    log4net.Config.XmlConfigurator.Configure();
}
Next I wanted to find a decent database appender that wouldn't affect the performance of our app too much. Luckily I found Ayende's AsyncBulkInserAppender  which, as its name suggests, is both async and queues up inserts at a configurable queue length.  With some minor tweaks, I was able to get this to work with our app.  I added some additional context to get our user, ala cookie from current request, and I could also stuff server variables into a custom column I created.  I started by overriding the Append event for the appender.  Inside that event you can add custom context to the logging event.

protected override void Append (LoggingEvent loggingEvent)
{
    try
    {
        SetUrl(loggingEvent);             
    }
    catch (Exception ex)
    {
        ErrorHandler.Error("AsyncBulkInserterAppender ERROR", ex);
    }

    base.Append(loggingEvent);
}

protected virtual void SetUrl (LoggingEvent loggingEvent)
{
    if (IsInWebContext())
    {
        loggingEvent.Properties["url"] = HttpContext.Current.Request.Url.ToString();
    }
}

private bool IsInWebContext ()
{
    return HttpContext.Current != null;
}

Next I added the appender to a few configs and set them to log errors only.  I found out while doing this that you can cascade configs within the same directory, even if they are in different app pools.  So I simultaneously cleaned up alot of our redundant web.configs during this process.  One thing you'll need to know is how to add a custom column to your appender.  Here is an example of the column I used to store the url.

<mapping>
    <column value="Url" />
    <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%X{url}" />
    </layout>
</mapping>

Everything was going well, and I was ready to build my interface.  I tested each site by throwing an error and checking the log, then I realized that SOAP exceptions from web services were outside the normal pipeline, and thus weren't caught within the global.asax.  Shit.  I did a little more googlejerking and hacked together the following:

public class SoapExceptionHander : SoapExtension
{
    public override void ProcessMessage (System.Web.Services.Protocols.SoapMessage message)
    {
        if (message.Stage == SoapMessageStage.AfterSerialize)
        {
            if (message.Exception != null)
            {
                log4net.ILog log = log4net.LogManager.GetLogger("WebService");
                if (log.IsErrorEnabled)
                    log.Error("An uncaught web service exception occurred", message.Exception);
            }
        }
    }
    
    public override object GetInitializer(Type serviceType) 
    { 
        return null; 
    } 

    public override object GetInitializer(LogicalMethodInfo methodInfo, SoapExtensionAttribute attribute) 
    { 
        return null; 
    } 

    public override void Initialize(object initializer){ } 
    
}

Add added in this in the web.config:

.......
    <webServices> 
      <soapExtensionTypes> 
        <add type="YourNameSpace.SoapExceptionHander,YourDll" priority="1" group="High"/> 
      </soapExtensionTypes> 
    </webServices>    
</system.web>

One thing you *need* to know, is that you can't test this from the little test page that .Net creates. The best way to do this is call the web service from a test page, making sure the service is throwing an exception. Don't waste hours of your life trying to debug why your custom SoapExtension isn't working. Argggg.


So now I've got all errors from all applications logging into one place.  I built my interface, with a filter on just about everything.  I also added the ability to 'handle' exceptions as a means of managing errors that need attention.



Much better.  Now we are depressed at the amount of log4net errors and warnings we see, but atleast we can address them. :)  Next on my list is the ability to maintain and push a branch of svn for 'hotfixes' so we can address these bugs realtime without rolling out code that isn't ready for primetime.

Nov 28, 2009

Adventures in Information Extraction (Part I)

I'm currently embarking on a personal project to parse and index content from certain specific blogs.  Before this adventure, my knowledge of the relevant body of information science was merely a term, Natural Language Processing (NLP), that I had no understanding of, and still don't.  Now that I've done some google-jerking, and with a bit of helpful clarification from a friend of mine enrolled grad school, I've hopefully started to define my initial effort into some more specific technologies and fields of study.  Nonetheless, alot of this will change as time goes on.

First let me describe, in general, my problem set and let me preface by saying, this is currently my 10,000 ft view of the problem approach.  I wish to target specific blogs and extract from them strictly blog content, minus the useless parts of the page (navigation, footer, header, ect).  Now, I hope to initially target two sites that contain lots of other blogs.  In doing this, I should be able to extract a general template (a common body for example) that contains the content of interest.  With content in hand, I would like to store this stuff in an index for quick retrieval using Lucene.Net.  Now what I am going to extract remains a bit muddy at the moment, as I try to explore the relationship between certain Natural Language Processing techniques and Lucene.  For instance, Lucene has the ability to tokenize (basically, find words), perform stemming (map variations such as 'traveled', 'traveling', and 'travel' to the root 'travel'), and filter stopwords ('the', 'a').  These things seem foundational to search in general, but what I really would like to index specifically are place names like 'The Taj Mahal', 'Pismo Beach', or 'Moe's Tavern'.  My research has pointed me to Named Entity Extraction, a subfield of Information Extraction.  Now the field of  IE starts to get a bit murky for me, and this theme will perpetuate as I go further down the rabbit hole.  I first started to try and find a good api and accompanying tutorial on these things.  Many roads pointed to NLTK and it's quite descriptive, and free, online book.  The library is built with Python, and I've traditionally been brewing C# and .NET for awhile.  But the idea of  learning a new language is welcoming to me.  My initial opinion of Python, is that its much easier to learn and use than a regimented procedural language like C# (though .NET has been making impressive headway into functional programming with F#).  Additionally, the syntax and libraries are more adept at parsing text.  The plan so far is to work my way through the online book to Chapter 7, which goes into exactly what I'm looking for.  Anyway, all this stuff is like drinking from a fire hose: new language, two new api's, and new subject matter, but I'm hoping I'll be able to put together a prototype in my spare time.  So far, I've been able to do some fun things with Python using Frequency Distribution (how many times a word appears in text), finding a word in context (collocation), and filter out some proper nouns using POS Tagging, a small step closer to getting my place names.  So for now, I'll keep inching towards Named Entity Extraction.  Once I have a working prototype for getting place names, I can circle back on some of the other things like feeding content into a simple Lucene index and figuring out the relationship between NLTK and Lucene.  Until I have some automatic extraction methods, I'm using Dapper to get me some data to play with.

It seems funny writing all this up, because tomorrow the direction will change.  But I'll stay flexible, and hopefully writing about my experience with help others avoid my stupid mistakes.

Nov 16, 2009

NHibernate 2.1 Performance Issues with Flush and LinFu

We recently discovered our nightly processes were taking much longer (50 - 80%) and wondered why. My focus was on the upgrade from 2.0.1.4 straight to 2.1.1. First thing I tried was to change the LinFu Proxy Factory to Castle. This had a surprising boost. For this particular operation some object properties were being accessed 100,000's of times. Redgate Profiler showed that, in aggregate, proxied object access can add up. 3300 objects loaded, with 3 many-to-one objects eagerly loaded, had some some properties accessed 350,000 times. One particular property being accessed took roughly 2 seconds over the life of the process, and some of the others took roughly a second. All adding up to roughly 12 seconds on a 100 second process. Whoa. Keep in mind that's with Redgate's deepest profiling on. Switching to Castle seemed to cut that to 5 seconds. Anyone know what's going on here?

But the big surprise of the day was what happened with flushing from 2.0.1 to 2.1+. A basic flush on this process on 2.0.1 took .42 seconds, but on 2.1.1 it took .79 seconds. Really? What the hell happened in between those versions? Just to be sure I checked 2.1 and 2.0.1, both were the same. I wonder if abstracting dynamic proxying from NH had some unintended consequences.

Needless to say we are back on 2.0.1 for the time being. This process that was taking 1 min and 11 seconds now takes 47 seconds. Why? A couple of reasons: first the process calls embedded business logic many times that ends up doing "safety" flushing when it's a mostly readonly session.  The next refactor is to fix that problem, but in all the flush increase difference accounted for 10 - 15 seconds.  Second, was the proxies which accounted for the remainder of the difference. I didn't do anymore benchmarks on 2.0.1 vs Castle on 2.1.1, but I'm sure there are differences there from all the testing that was done. In either case there's not anything in 2.1+ that's worth these performance tradeoffs we experienced. NH.Linq seemed like it would have been a good move, but it's not yet ready for production use. So for now I'll take the perf benefits of 2.0.1 for the minor upgrades of 2.1+.