MicroStrategy’s Enterprise Semantic Graph: Is it alive ?

This article’s eye-catching title may look like clickbait – but, having worked for many years with the product, I’ve come to ask myself if a large, automated, system consisting of many components interacting with each other could be described as alive. After all, in the rare circumstances when the Intelligence Server conks out, we often refer to it as dead – and for that to happen, it had to be alive in the first place, no ? it can even resurrect itself, which opens up a whole new avenue of debate…

Before we proceed, a disclaimer and some acknowledgments:

I am writing this in my personal capacity. While I work for MicroStrategy, I am in no way describing any future functionality that I know about. This is purely a theoretical exercise, for my benefit and that of my few readers.

I have taken advantage of ChatGPT to ask questions, and of DALL-E to generate some of the images in this article. The rest of the text is my own work.

Let’s define a few things

It’s 2023, and I have taken a habit of asking a famous AI what it thought of interesting questions. So here we are:

I particularly like ChatGPT’s first sentence: MicroStrategy’s semantic graph is not a living entity in the biological sense. Let’s ask what determines if something is alive:

That’s interesting. I would say that a MicroStrategy installation can meet some of these requirements. How much is met depends on debate, and your desire to make facts fit a hypothesis. Without thinking too much about it, the match is good on organisation, homeostasis and response to stimuli. You could cheat and say that, through symbiosis with humans, it can also achieve growth, adaptation, metabolism and reproduction.

So what would it take to bring it to life ?

A while ago, I amused myself by modelling what a synthetic personality would look like. The image above shows a high-level view of Cynthia. The lower levels exist, but I am not sharing them yet :). Cynthia has some aspects which if created could be added to MicroStrategy to give it agency. Whether you would want to do this is another matter… I reckon that a team of decent game programmers could achieve this.

But the article talks about the Semantic Graph ! What about it ?

Many advanced lifeforms have a nervous system that stores memories and knowledge, responds to stimuli and takes action, and develop, through learning and evolution, purpose and motivation.

I suggest that, were a MicroStrategy installation a living thing, then the semantic Graph would be its nervous system. This nervous system evolves, and knows about its environment, not only through a mixture of symbiotic impulses from users, architects and administrators, but through its own sensorium via diagnostics and Platform Analytics. It is capable of translating data into information, for now through human configuration, but eventually I see it using an artificial intelligence adjunct.

So what is the Enterprise Semantic Graph ?

There are many ways to explain this. For me, it’s a member of a trinity of domains: You have the software, the data and the Enterprise Semantic Graph. It’s an ordered, connected collection of objects that describe and govern the configurative and transformative processes that the software uses to transform and present the data.

The image above shows a very simple semantic graph. Below is rather more complex one:

Again, this is a tiny portion of an Enterprise Semantic Graph. For those who know MicroStrategy, the term Schema is often used as a substitute. But that’s a simplification. Limiting the ESG to a collection of tables, attributes, metrics, facts, filters, reports, dossiers only gives you part of the story. I affirm this because I have spent quite a bit of time trying to visualise the ESG in a way that is easy to comprehend and to navigate. And the more I tried, the more I realised the following:

  1. It’s too big to encompass in one go.
  2. I am missing a number of dimensions to really understand how a system behaves.
  3. The visualisation mechanisms, out of the box, and even looking further afield in Python graph visualisation libraries, are inadequate to perform the task correctly.

Which is a shame, because if I wanted to fit MicroStrategy with an AI personality module, I would really need to research in depth the pathways and patterns that make up the ESG.

Let’s look at the above points, starting with point 1. The ESG, for a mature installation, is enormous. You can easily reach tens or hundreds of thousands of objects and their interconnections. With point 2, you need to enrich the ESG with the following:

  • Platform Analytics data
  • Subscriptions and schedule events
  • Telemetry from the underlying systems
  • Groups and their users, and the networks they form across projects
  • I’m not even sure the list stops there !

And, finally, point 3. How do you visualise such a complex network ? You can only ever see a portion of it, it’s like looking at a landscape through a keyhole. However:

In my younger days, at university, my final year project was a VR world generator that used BI and MIDI data as a basis for objects in the world. But now we have AI, what I am tempted to do is to take the ESG and all the other data highlighted in point 2, capture the relationships, tokenise them and ask an AI to build me a virtual word with it that I could explore.

Would the world look like this:

Or would it look like this:

I am not sure such an AI exists. Yet. Furthermore, it would seem a complete departure from the theme of this article, if it were not to confirm that there is enough richness and depth in the ESG to achieve complex, living (in the cybernetic sense) structures, be they worlds representing their complexity, or minds managing the MicroStrategy installation.

A call to action

We live in times that I used to read about in science fiction books a few decades ago. With advances in AI, I believe we should go full speed ahead and either invent a way to navigate MicroStrategy as you would in a game like No Man’s Sky, or that you should build and train an AI to administer a MicroStrategy installation. Give me a few billion and I’ll do it.

Visualising the Gambler’s Fallacy, or how to (not) win the lottery

Magnus D from London, United Kingdom, CC BY 2.0 https://creativecommons.org/licenses/by/2.0, via Wikimedia Commons

I have played Euromillions for many years. It’s been an effective way of turning money into nothing, and I can only justify it by the warm feeling that I am contributing to good causes. I have won something many times, mostly single digits… But one day, maybe, I could win a life-changing jackpot. Which I will not win if I don’t play.

Up until now, I have never really thought about numbers I have played. I vaguely know that you have as many chances of winning using the Lucky Dip method as you do with a careful selection of numbers. It’s also said that you have more chances of being struck by lightning. And then, there’s the Gambler’s Fallacy:

The gambler’s fallacy, also known as the Monte Carlo fallacy or the fallacy of the maturity of chances, is the incorrect belief that, if a particular event occurs more frequently than normal during the past, it is less likely to happen in the future (or vice versa), when it has otherwise been established that the probability of such events does not depend on what has happened in the past. Such events, having the quality of historical independence, are referred to as statistically independent. The fallacy is commonly associated with gambling, where it may be believed, for example, that the next dice roll is more than usually likely to be six because there have recently been fewer than the expected number of sixes.

https://en.wikipedia.org/wiki/Gambler%27s_fallacy

But still, I wonder if there is a way of picking numbers, a pattern that would make me more likely to win the jackpot ? The Gambler’s Fallacy says there isn’t a way. It would be interesting to research this, however, by getting hold of the draw history and analysing it.

Finding the draw history

The official National Lottery site lets you download draw history, but not in a form that’s easily consumable. I have found this site:

http://lottery.merseyworld.com/cgi-bin/lottery?days=20&Machine=Z&Ballset=0&order=1&show=1&year=-1&display=CSV

which provides data in this format:

I can copy and paste this data into a spreadsheet:

Which I then import into MicroStrategy:

I now have a dataset in a form suitable for analysis. A draw consists of five numbers (N1 to N5) with values ranging from 1 to 50, and two additional numbers (L1 and L2) ranging from 1 to 12. Each row represents a draw, and is identified by its draw date. We have also the day of the week on which the draw occurred.

Visualisations

Ball Stats

This is a classical way of showing data, looking at each ball in isolation, with a trend for the ball value over time.

Click on the images to view in full screen.

This is not really providing guidance. You can see that for each ball, over time, there is no predictable pattern. This visualisation is not helping me pick numbers, so far.

Ball Trends

A slightly different treatment, with a more visible timeline of ball values over time:

This helps a bit more, in that you can get a ‘feel’, for each ball, of values that seem to occur more than others… with absolutely no rational, scientific or statistical validity.

Histograms !

Now you’re talking… This seems to suggest the best value to pick for each ball.

And this where I had an interesting insight: What if there was a relationship between the value of one ball versus another ? This though occurred to me when, using an info window, I clicked on the value 2 of the histogram for ball number 1, I noticed that ball number 2 had the value 10 repeating on more than one line. Exciting !

But how to visualise this in MicroStrategy ?

Sankey to the rescue

Sankey diagrams confuse me, because I have struggled to find a use case for them that didn’t require some serious thought to interpret. But in this instance, it was the perfect method to visualise the relationship between a pair of numbers repeating across the set of (n1,n2,n3,n4,n5,l1,l2).

And there, magic seemed to happen:

But not quite in the way I intended. The first image shows the Sankey across a limited set of draws. In this, there are clear pathways from one number to another… so picking these should make me more likely to win, if it weren’t for that damned Gambler’s Fallacy. Another way of interpreting the Fallacy is that the lottery is a system without a memory – it does not remember, or is not influenced, by what has happened before. What happened in the past is irrelevant,

This is demonstrated by the second picture, which is using a far greater set of draws. You can see the lines linking one number to another are much thinner, and that over time, any visible, distinct pathways should disappear altogether.

Is this beyond hope ?

When it comes to predicting, or even selecting lottery numbers, then it’s pretty much impossible. It’s a physical system, so even if you had the same machine and the same set of balls (it’s not the case, many machines and many ball sets are used in many permutations) that would somehow develop a bias over time, you’d only get a reduction in the odds, still astronomical. So you’d be more likely to guess the numbers, but only more likely.

Other games of chance have better odds, and can be beaten. There are a good number of texts and books out there on the subject.

For me, though, I am happy that I have managed to get my head around the Gambler’s Fallacy by visualising it in MicroStrategy.

Going further

There’s science and reason, and then there’s faith and hope. What I have there is a fine system to store and analyse draws. I have thought of ways to enhance this:

  • Scrape the source site every week to automatically update the dataset.
  • Use Python and mstr.io to implement various pseudoscience algorithms (Numerology, horoscopes) to pick auspicious number sets.
  • Analyse patterns based on the sum of all numbers, or the intervals between them.
  • Track the machines and ball sets used, and also the draw sequence of each ball. See if a bias emerges…

To close, I’d say the fun of doing all this in MicroStrategy mitigates the disappointment of forever not winning the jackpot.

Rockets, Volcanoes and Goodbye to 2020

This may surprise you but I did manage to travel a bit during this fateful year. There were three main trips: Portugal, America and France, and they mark also the progress of the pandemic, from blissful ignorance to a vague anxiety, then full-blown mayhem.

Portugal, January 2020

Frozen Dusk in Northern Spain en route to Portugal

We drove, in a Land Rover Discovery, all the way to Braga in Portugal. A relative of mine was relocating there and this was the big move.

We set off from Hemel Hempstead, drove the Tunnel and headed directly for Bordeaux, bypassing Paris. We overnighted in Bordeaux, then drove to the North west of Spain, stopping there for a night. We completed the journey the next morning.

I flew back to the UK from Oporto.

Impressions

The majesty of the mountains, the emptiness of Northern Spain, the cold, the resonance of Portugal in our family landscape.

America, February 2020

Awesome ! Just awesome.

I visited there in early February to speak at the MicroStrategy World conference in Orlando.

Impressions

Harry Potter World, Cape Kennedy Space Centre. Warm weather in February, the Pandemic moving from the background into our lives.

France, September-October 2020

Le Puy de Dôme, dormant apparently…

I travelled to France to work on a major upgrade for a MicroStrategy customer. This was fully Covid-constrained travel – masks, second waves and all.

Impressions

Absurdly cheap train travel, extinct volcanoes, wonderful food and people, wearing a @£&** mask all the time.

Empty airports and planes, lost luggage being a blessing of sorts, near failure of travel plan.

The biggest journey

2020… A year like no other !

The brutal crimp imposed by the pandemic is the biggest adventure for us all .We may have been confined but everything else has moved around us – and we have discovered useful things about what really matters, what we can do without and what we take for granted.

And what about 2021 ?

I have a single first class return from Paris via Eurostar. I was going to go to Paris before Xmas but have managed to move and rebook my ticket for late February. It’s now a matter of getting over to Paris somehow, and hoping that Eurostar still operates – the Pandemic and Brexit have made it fairly challenging. Such fun !

Anyway, have a great and happy New Year.

Revisiting Clausewitz: The Picard Quadrant, a tool to measure engagement

By Enterprise-D_crew_quarters.jpg: Derek Springer from Los Angeles, CA, USAPatrickStewart2004-08-03.jpg: Cdt. Patrick Caughey[1]derivative work: Loupeznik (talk) - Enterprise-D_crew_quarters.jpgPatrickStewart2004-08-03.jpg, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=17031867
If you’re not a Trekkie, Captain Picard is famous for his ‘Engage’ directive, hence my tribute to him in this article.
Remember the Clausewitz Quadrant ? I wrote an article about it, you can see it here.

The thing with quadrants is that you can bend them to many uses. In the above-mentioned article, I express reservations about their value if you do not investigate the methodology and the data that they summarise.

As you know, I have been writing a series of articles about employee engagement, and I am of course very much behind with the latest and hopefully final instalment… But as a side project, it came to me that you could re-purpose the Clausewitz Quadrant to classify individuals into categories defined by their industriousness and engagement, or lack of it.

So here goes:

The Picard Quadrant – the axes

Picard quadrant

The X-axis is the same as the Clausewitz, ranging from Lazy to Industrious, but in the engagement context:

  • Lazy means an aversion to working hard  for hard work’s sake.
  • Industrious means a strong determination to accomplish tasks and acquire new tasks as they become available.

The Y-axis now replaces Intelligence with Engagement. In many occupations nowadays, a modicum of intelligence is an entry-level requirement, so this measure is no longer appropriate. The axis now ranges from Indifferent to Engaged:

  • Indifferent means not caring at all about the big picture and simply doing the job, no more, no less, with little regard to participation in the organisation’s mission.
  • Engaged means looking beyond the boundaries of one’s position in the organisation, working to achieve the mission and to grow one’s career in synch with the progress of the wider team.

Let’s now look at the four quadrants in turn.

1:Indifferent and Lazy

Indifferent-lazy

The Lazy and Indifferent category, in the private sector, is an endangered species. Efficiency pressures do not make a favourable environment for such individuals.

They will tend to work in areas that are either automated or  outsourced. The increasing privatisation of the public sector is also narrowing the field in which such individuals can exist.

This means that, generally, it becomes important to find an occupation where the individual actually cares for, and thus engages in, the organisation that employs them.

2:Engaged and Industrious

Engaged-Industrious

Being Engaged and Industrious is an absolute gift to an organisation. Many tasks can be given to these people, and they will thrive on the challenge.

Furthermore, they are keen to take on more work but run the risk of burning out.

The astute leader will know how hard to drive and challenge such people to get the best sustainable yield.

Such people can often over-commit to deliverables and may be disappointed when colleagues or events do not follow their pace.

3:Engaged and Lazy

Engaged-Lazy

Being Engaged and Lazy is an attribute that should be valued providing an organisation is not entirely composed of such persons.

People who seek to get the best yield out of the lowest amount of work possible will look for efficiency and simplicity in order to get the tasks done. They will also have bandwidth to look at the bigger picture and may provide or enhance strategic decisions.

As such, they can also think laterally and reach innovative solutions which would elude their more industrious peers.

They do need to be led and motivated to keep the lazy side of their nature under control. Promotion is often a method used to get the best out of this category.

4:Indifferent and Industrious

inidifferent-industrious

This is quite a difficult category, if the person in question is at a lower hierarchical level, or plays a key part in a process at any level.

The typical behaviour is one where there is competency and knowledge, but solely in the execution of a defined set of tasks without any regard for the welfare and evolution of the organisation or the colleagues.

Such individuals do not welcome change and will be inflexible when thinking out of the box is required. This is increasingly a toxic attitude in today’s workplace and should be remedied by appropriate coaching.

So does this quadrant have any value ?

I think it does. As with the original quadrant, it’s a blunt tool, using axes whose values are almost impossible to measure. Its purpose is clear, however: To illustrate the value of engagement and the varying ways in which engagement benefits an organisation and its individuals.

Crucially, it also highlights the danger of having a disengaged workforce. This must be remedied – by the organisation’s leadership applying relevant coaching, or by the individual by finding an occupation where engagement will occur.

Increasingly the world of work will consist of leadership, ,supervisory or creative tasks. Processes will be automated by robots, machine learning and AI systems. Engagement will become a vital aspect of an employee’s value, for work to have any meaning beyond a simple means of subsistence.

This article is relevant to the next one in the works, where I describe an organisation where engagement is inevitable, and supported by numerous technologies, including, of course, MicroStrategy.

 

Of the relevance of culture in modern organisations and its role in employee engagement

1024px-Assyrian_orthostat_in_the_Pergamon_Museum,_Berlin
Is it possible that excessive hair and beard grooming may have led to the fall of the Assyrian Empire ? 

This article is the second in a series leading up to a potential corporate cultural revolution enabled by one of the key technologies created by MicroStrategy, my employers. In this article I set up the importance of culture and value systems leading up to employee engagement. 

I like studying history. In particular, the rise and fall of empires. There’s so much of history that crosses over into modern organisations, and yet so many lessons of history that do not get learned. I am taking the bold step of comparing corporations to civilisations, a valid one because both of them are human artefacts.

Culture is better than strength

I think it is evident that for a civilisation to last, to be remembered, to be influential, a strong culture is key. Examples abound: Look at the influence of Greece and Rome on the development of the West, the contribution that the Arab world made to science and medicine, the absorption of Kublai Khan’s Mongols by the apparently subjugated China, the survival of the English language  in Great Britain despite the French-speaking Normans. During my exploration of this topic,  I have noticed that purely military civilisations do not last as long as those with a powerful culture.

A powerful culture, supported by writing, laws, art, religion, can last way beyond the physical presence of the civilisation that gave birth to it.

How culture fares in a corporation

But is that a desirable feature for a modern organisation ? In fact, in these days of re-branding, of mergers and acquisition, it seems to me that corporate culture is an ephemeral thing, and sometimes culture is seen as an obstacle to the functioning of a free-market economy. How many times do you see the word ‘culture’ used in negative terms, being associated with resistance to change or deleterious practices ? For the modern corporation, a shift in culture is often seen as a key way to effect change. Whether that change is to benefit the shareholders or the employees (usually the former) is a subject for another article. Whether this culture change is carried out with regard to its impact on people is also another matter…

Printing-press_-_Buchdruckerpresse_(2311719869)
That invention caused a singularity – knowledge became affordable and universal. Sounds familiar ?

Change is inevitable

And yet, moving back to the domain of civilisations rather than corporations, it’s important for a culture to adapt with the times, to withstand shocks such as natural disasters, scientific discovery or religious and political upheaval. What matters is that the civilisation emerges from the metamorphosis enhanced, and ready to prosper and expand under new parameters. Maybe this is what happens with corporations, except that what I find missing is a level of continuity that could foster a sense of belonging by those who take part in it.

What is a culture ?  It  is, until we can understand whale-speak, a uniquely human artefact. Put simply, it is a combination of shared history, language, values and creations that underpins a civilisation. A corporation also has these components. I suggest that where a corporation and a civilisation part ways is in the purpose: A corporation has a defined goal: Make things, make money, create value and so on. A civilisation’s purpose is, again simply put, to endure.

Corporations do not endure. Or if they do, they no longer really resemble the original state they were founded in. This may be because the external forces that are applied to a corporation force change at a far greater pace.

Is this a good thing or a bad thing ? it all depends on the perspective. For the owners, or the shareholders, it’s essential that the vehicle through which they derive their wealth can adapt to changing environments. For them, the cultural metamorphosis is a good thing.

For the workers, it’s often a bad thing because a rapid change of purpose where elements of the corporate culture might disappear can cause disengagement and loss of motivation, or worse a loss of employment.

Just to get this in: I particularly abhor the concept of “business process engineering” because it is a thinly veiled euphemism for large-scale job losses. It was a concept that emerged in full in the 1980s, and many of the texts I have read on the topic gloss over the human element. It’s very ‘last millenium’, don’t you know. 

bpr
The job-cutting machine, aka the Business Process Re-engineering mechanism. The favourite tool of the sociopath, in my opinion.

When the jobs start disappearing, and the shared history fades away, it might be seen as a necessity by the leaders and the owners, but this does not mitigate the loss felt by those whose purpose has ceased to exist.

Sadly, we have to accept the inevitable – corporations that cannot change are eventually doomed. So are civilisations, although in this case their culture might survive. I don’t think you can say the same about corporations.

Shared Values

1024px-Eleanor_Roosevelt_UDHR_(27758131387)
An important set of values. Often ignored…

Before I finally talk about employee engagement, I need to talk about shared values. As I said previously, civilisations and corporations have a common component, a set of values or behaviours which, in the case of corporation, should support its goals and its purpose. How these values help the achievement of the purpose has to be signed up by all in the organisation, in total transparency. Fail to do this and cynicism about the leadership’s intentions will spread like a nasty stain through the corporation’s morale.

Workers should not accept toiling to enrich the leaders or the owners without a fair reward for their work, and an understanding of its value for the organisation. They should also not accept a set of values that are not endorsed by all in the organisation.

Chaplin_-_Modern_Times
The worker as a cog in the machine. It’s had its time…

This may sound a bit socialist to you, and it’s certainly an aspiration in many cases. But in our coming world of automation, the commoditised employee is an endangered species… I see it everywhere: repetitive, predictable tasks are increasingly performed by robots or proto-AI devices.

The human element is no longer a cog in the machine – it becomes a supervisory agency that guides and controls the machines to deliver a purpose or an outcome. Again, the social impact of automation is another rabbit-hole, nevertheless it is crucial that, in their new supervisory roles, workers and leaders share the same values, and perforce the same engagement.

Employee engagement, at last…

So what does employee engagement actually mean ? Is this another unreasonable ask by leaders and owners: We don’t want you to simply work for us, we want you to invest yourself in us. 

This is a big commitment – instead of seeing employment as a means to earn an income, strictly bounded by working hours where the employee switches the work mindset on and off on a defined basis, engagement means that the job requires an elevated  level of focus closer to what is normally reserved for raising a family, or pursuing a cultural or sporting activity. The employee needs to care, and will probably think about work outside of the normal working boundary.

This is fine if it works both ways: the employee has to have a clear reward path that is congruent with career aspirations. It’s a matter of strategic alignment: if this is not the case, then one party or the other does not benefit from the relationship, and an adjustment is required.

The Chief Culture Officer

Yet another silly title, you might think – but actually, if there is one hat amongst many that a Chief Executive must wear, defining and promoting a corporation’s culture might be the most important one. Done right, it provides a value framework where employee engagement occurs naturally. If I may refer to one of my previous articles : Leadership Observations: The Resistor-Capacitor Theory  , this cultural messaging is of paramount importance, given the right balance.

Closing off – for now

I could ramble on for pages and pages about this. So it’s time to pull together the loose strands of this article and lay the groundwork for my definition of employee engagement:

Given a good cultural fit, and a realistic set of values that are followed by all, it should be possible to nurture employee engagement providing a strategic alignment exists between the goals of the employer and the employee.

My next article will describe one of the methods by which employee engagement can be fostered. And yes, it will be time to pay tribute to my esteemed employers because I will use a key technology of theirs to enable this method.

Spice Cupboard Rationalisation, and thoughts about corporate efficiency and employee engagement

spice
My rationalised spice cupboard. Free space ! Amazing !

This is the first article in a series concerning employee engagement. In our rapidly changing society, it is becoming vital that all who take part in the organisation understand what the objectives and incentives are at all levels. Increasingly, our work is turning into the processing and enriching of knowledge – but knowledge of the system in which this happens is essential, and this is where employee engagement comes in. In this article, I explore the causes, impacts and outcomes of corporate reorganisation, and how engagement at all levels is key to make such exercises less traumatic.

I love to cook. I have a work area in my kitchen, organised so that all the necessary elements for the production of various dishes is as efficient as possible.

i use spices and herbs in my cooking. Over the years, and much experimentation on my family, I have worked out the ideal blend of these ingredients to produce results which are sometimes restaurant grade. There have been disasters, such as the Chernobyl Green Curry or the Singapore Sludge. But my family have survived, so that’s all good.

Anyway, to the point of this post. I have a spice cupboard and it was a complete mess- empty jars, spices in duplicate, ingredients never used resulting in an endless search for the items which I needed most. Annoying and frustrating !

So I carried out a reorganisation. I was ruthless – I binned all the ingredients I have never used, merged some jars and rationalised the layout of the ingredients so that the most frequently used ones were easily to hand. The result: blessed efficiency and reduced frustration when preparing food.

This made me think. I have spent many years working with organisations that, on a more or less regular basis, went through reorganisations. I wondered if the process I used to reorganise my cupboard was similar to the one used when a corporation goes through a rationalisation exercise.

Humans are not spice jars. If my spice jars had feelings, then what I had done would have been cruel, cold and ruthless. But at least I was quick. When a corporation does this, the process is nearly not so fast, but the impact on those targeted by the reorganisation is considerable. The effect on morale is huge, and repeated instances of these happenings have a deleterious effect over time.

Another side to this is that there are times when reorganisations are essential. Again, looking at large corporations, these acquire over time a cohort of people whose functions become unclear or redundant. The onset of automation increasingly replaces swathes of workers, and you cannot help but feel that the corporations actually perform a useful social task by providing employment to people who might generate little value, and sometimes are a cause of cost and friction.

This is all well when times are good. But when the organisation is under stress, efficiency becomes key because survival is at stake, and areas that are not core to the purpose of the corporation must be looked at.

And thus, rationalisation takes place. Some organisations do their best to make the process as fair as possible. Some don’t. I have heard some horror stories about workers being made redundant by text, or a climate of fear and doubt caused by poor communication.

I’d conclude that it’s important, either as a worker or as a leader, to be as engaged as possible in the organisation. This is not to promote a culture where every body is super enthusiastic (that would be tiresome for us Brits), but more to make sure that the purposes of the organisation are well understood and communicated, and that the challenges, internal and external, are taken on board so that all can make the right strategic decisions – the leaders about the future of their organisation, the workers about their role and their relevance to it, and ultimately their career moves. Denial is not an option.

Just so you know, my ultimate goal is to promote data democratisation to assist employee engagement. It will favour the platform that my current employer produces, so I am not entirely without bias in my writing. Still, I hope that the journey I take to the eventual conclusion is one that you will recognise, and maybe act upon. It’s up to you.

 

Leadership Observations: The Resistor-Capacitor Theory

Resistor capacitor
The resistor and capacitor symbols. When you write about business, you can press-gang other domains for some contrived metaphors. Sad, but true.

The good thing about a blog is that you can write about stuff you experience and observe, but which might not be correct because it’s just your recollections and perceptions, there’s no deep research and no financial or professional motivation.

One such thing I want to talk about is leadership. Many, many books, courses and articles exist on the topic and I have read none of them ! But I have worked in organisations all my life, and not being a senior exec or a CEO I have been the recipient of leadership in a variety of forms. I also observe the leadership styles that members of my social circle experience, and there are a few interesting strands that emerge from this lazy yet curious study.

One such observation is about the role and type of the top person of an organisation, the CEO. I find that there are two main categories for this person: Driven, visionary leaders with absolute belief in their missions who direct their subalterns with intensity; and detached, discrete individuals who leave the leadership message up to their immediate reports.

Let’s call the combination of the CEO and immediate reports the Leadership Engine. This construct guides the organisation and executes strategy. Depending on the class of leader, the subalterns will behave in one of two roles:

The Resistor: When the leader is very strong and intense, the subaltern will need to have  the capacity of translating the blast of leadership messaging into guidance and instruction that can be executed by the underlying reporting chain. Without this message damping system, the lower ranks of an organisation will not function effectively as they will not be able to withstand the force of the message. I call these message-attenuating subalterns resistors.

Resistor
There’s always a risk of subaltern burn-out, unless they have sufficient talent and emotional intelligence to handle the leader’s intensity.

The Capacitor: When the leader is less hands-on, more theoretical than practical, the subalterns will need to provide the leadership strength that might be lacking. These subalterns become leaders in their own right, and the executive team becomes more like a federation of fiefdoms with the CEO as a ceremonial head.  I call these message-amplifying subalterns capacitors.

Capacitor
This set up can lead to competing fiefdoms and abuses of power. Also, there may be a need for resistor-type people lower in the organisation chart. 

In my experience, a resistor style leadership team is strong on execution but weak on confidence in the lower ranks and may stifle the innovation and engagement flow across the organisation chart. It may not be capable of challenging the leadership message which could lead to poor decisions lacking a critical evaluation.

A capacitor style leadership team will perform well within the functional streams, but may become inefficient or even dysfunctional if the capacitor coordination is poor, or adversarial.

Each style has its advantages and its downsides. It is down to the leadership to recognise where the strengths reside, and where counter-productive behaviours need to be mitigated by accepting a constructive critique. When you consider the type of individuals that can be found in leadership teams, this is not always an easy task.

And yet, this is increasingly important, because the most important asset in a modern organisation is the talent of its workers, at all levels. Tuning the leadership style to get the best sustainable yield is an essential, ongoing task.

So which type of leadership engine works best ? This is where some more serious research would be needed. Ah, given infinite amounts of time, what wonders we might reveal (probably not).

And that’s where my thinking stops for now. I am, after all, merely a person who plays with data for a living, not a management consultant. I have had the good fortune of encountering effective leadership in my career: I would not hang around very long in a company that didn’t get that right.

Election 2019: A Dossier about Marginal Constituencies

Marginals2017
A nice map of UK constituencies, where marginal constituencies  (majority of less than 5%) are shown in red.

Damn that election!

I was supposed to lovingly polish my video about Crime in the Home Counties, and then our government decided to call an election in less than six weeks time.

Some distraction, I can tell you. It’s a distraction because I have all the data I need to explore the various battlefields where nothing less than the future of the UK is being decided. Interesting times…

Luckily, I have the data ready, in a data import cube. I shared previously a map of who came second at the 2015 election, so I know the data set in detail. Thus, with a new dossier and my trusty custom constituency shape file, I was ready to find out which constituencies had Members of Parliament whose majority was the thinnest.

Hand on my heart, I swear that the dossier you are about to see took me no more than 15 minutes to put together. This is data exploration at its finest.

So, below, I have zoomed in on the northern Home Counties. The areas in red are those where the majority of the winning party at the 2017 election is the slimmest.

Marginals2017NHC
I know, there is a lot of the country to look at – but I live in this area, so there.

Now, for no other reason that this is of great interest to me, I am filtering the visualisation to only show areas under Conservative control:

Marginals2017NHCCons
By area, the Conservatives appear to control quite a bit of the country. Most of the population lives in towns and there, it’s a different story…

Clicking on an area shows you a tool tip with relevant information. And, of course, a Hyper card with crime and deprivation about the said area ! This is a bonus enrichment which can add a serious amount of instant insight to your discovery session…

Marginals2017NHCHyper
Hyper Intelligence ! It’s magic !

For additional info, I have added a linked grid showing the track record for the selected constituency.

Marginals2017NHCConsDetail
Erm – I hovered over another area so the majority is not that of the highlighted area. My bad.

I am not a psephologist. But looking at those marginal constituencies, and there are a number of them around the countries, you can see that factors such as turnout and, apologies for swearing, Brexit, are going to make this election very interesting.

At the very least, I have a tool that helps me put some context about the key battlegrounds where this election is going to be fought. If I find anything interesting, I’ll share it, while doing my best to remain politically neutral.

HyperIntelligence, Crime and Forthcoming Videos…

I am getting mobilised slowly  to start building demonstration videos of the amazing (to me at any rate) stuff I am doing with my social and crime data, and my freshly upgraded MicroStrategy 2019 system.

There’s a lot of moving parts. Working out and acquiring the best microphone to record voice, the screen capture software that does not break the bank – or require  a 15 grand Mac to operate… Then there’s getting the new Microstrategy components ready – a demo account for my project (can’t run everything as administrator in real life, I want to show what a consumer experience looks like), polishing the dossiers and cards, checking that the story and the dossier chapters hang together correctly. Plus there’s the copyrights and legal things to consider, and I have to be careful that my ad-libbing does not stray into contentious territory, so I am scripting – loosely – my upcoming demos.

The thing is, it’s fun. I wouldn’t do it if it was tedious… I have two core values. The first one is “Neither bored nor ignored”. The second is “Only if I have to”. It explains why I am not a CEO, I suppose. So this video demonstration endeavour respects the spirit of my values. It’s not boring, I have a very small but precious amount of likes on my posts, and I want to do this. I am not being forced, on the contrary.

Still, it’s taking shape. First, i have an ident prototype for my upcoming channel (where I will host it is still a mystery):

 

It’s a prototype, testing the screen capture. The fractal tree is generated by hippiefuturist.

The title is a working title. I have a couple of secret weapons: My brother and one of my sons. The former, musician and composer,  is not aware yet that I am going to ask him to compose a title track, and some background music. The latter is doing a digital media degree, and is going to polish up the various offerings I aspire to produce: Podcasts, demonstrations, interviews, outside broadcasts… all in good time.

The first offering will be a demo of MicroStrategy’s HyperIntelligence, leading into an exploration of Crime in the Home Counties – see Crime in the Home Counties: A data discovery journey – Part 1.

Here’s Hyper working in Chrome and a bit more slick:

HyperNice
In Google Chrome, I can search for the towns around me, and I get instant social and crime context.

Here’s the dossier about the crime exploration in the Home Counties:

Dossier
Luton, a troubled town – with Hyper working inside the dossier, for instant comparisons !

I hate promises, because I don’t like breaking them – but I have the ingredients almost where I want them, so I just need to make the time to produce my first demonstration video. It’s imminent!

I have a lot of data stories to tell…

Crime in the Home Counties: A data discovery journey – Part 1

The Premise

Crime is an emotional subject. It is constantly being mentioned in the media, more or less stridently depending on the publication, and more or less correctly. As a “citizen data scientist” (actually someone who needs to go the gym but prefers mucking about with data), it is always my desire to verify what the media, and the politicians, have to say about the subject.

For this study, the task is made possible by the Home Office publishing crime statistics on a monthly basis, by police authority and down to local level.

You can get hold of this data here:

UK Police Data Site

The purpose of this article is multifold:

First, it aims to demonstrate how a tool such as MicroStrategy can be used to bring to life data about a given subject. Other tools exist, but as I have stated in previous articles, I work for MicroStrategy and I am unashamedly confident both in the abilities of the software and my competence with it.

Second, I am using the output of the first premise above to answer some initial questions: What does crime look like in the major towns of the Home Counties ? Does the data correlate with what I know of the towns ? What sort of crime is prevalent outside of urban areas ? What are the outcomes of police investigations into the crime, which area and which crime types have the best resolution rate ?

Third, can the output of previous two premises be used to build a compelling story that could be published in its own right ?

That’s the scope of my current investigation. However, I can already see ways of transforming this exploration into a powerful tool, given the right investment.

For instance, what enrichments could be possible to put the data in context in terms of deprivation, population densities, political prevalence and ethnic composition ?

Finally, and that is the true prize for me, will I be able to build, once I have upgraded my system to 2019.2, (eventually, I keep being distracted)  a HyperIntelligence capability which will provide me with a pertinent summary about a given location in the Home Counties whenever they pop up in my Chrome browser ? Will I be able to use the Police API to keep the data fresh, and will I be able to scale up my solution so that it covers the entire country ?

About the Data

Getting hold of it

The link at the start of this article will take you to the initial download page. You can download all of the data, however this is a colossal amount and it would take a great effort to get it all in a shape ready for investigation.

The site helpfully provides a query tool permitting the download for a range of months and police authorities:

Custom police data download

Datadownload
Select your time range and your police force

For this article, I have selected a year’s data from February 2018 to February 2019, for the Thames Valley, Bedfordshire and Hertfordshire police areas.

This utility will then squirt a zipped folder to your Download area, which you can unzip into a location of your choice. Your downloaded data will be structured thus:

unzipped
Yes, a folder per month. Some stitching together in perspective…

Each folder contains these files:

Foldercontent
We’re only using the files containing the word “street” for now.

Looking at one of the files via Excel:

Filestruct.JPG
Good old Excel.

We note some interesting features. We have longitude and latitude that plots the rough area where the crime took place. The data is anonymised so the precise location is not given. We have the LSOA code, which is a census geographical area representing roughly 1000 people. We have the month, and we have a crime ID. You will notice that some crime IDs are null – these lines represent Anti Social Behaviour Orders, which are classified and resolved differently. These nulls can prove troublesome when we manipulate the data, so we’ll need to do something about it.

Staging the data

You’ll have noticed that our data is delivered in pieces – we have one csv per month and police area. We could load these chunk by chunk into a MicroStrategy cube, however this would prove to be cumbersome and lengthy. In addition, we want to future-proof our efforts so that our data can be manipulated and enriched in later stages.

So for this exercise, I have imported the data into a MySQL database. In there, the data structure looks like this:

mysql
You can create the table structure beforehand and import the data into it, or you can take your chances and let MySQL create the table, with its best guesses regarding the column types. Your choice, you could be lucky.

Now that our data is all neat (well, almost) and tidy in a MySQL table we can point MicroStrategy to it and import it into a cube:

import

And in due course, we have a cube upon which we can build a dossier.

Data Import Dossier

That’s a great place to start exploring the data and to get a feel for the sort of visualisations that will tell the best stories.

However, in its current form, there are a number of useful data features missing which would help greatly with the investigation that I require. I need:

  • To group all the LSOAs for the towns that I want to compare. I can create a group in my dossier, and this can help me work out how the grouping will work.
  • To count the total number of crimes by LSOA, crime category and month and have this as a discrete number ( This will be a level metric).
  • To work out the percentages of the outcome within the Crime category.
  • To only have the data in the cube for the towns I require.

I can do all this in  a dossier as those features are supported. However, this all represents a fair amount of analytical processing – if I add the use of maps in there, this makes the dossier not as fast as I would like.

That’s fine, because it’s a necessary step on the path from exploration to exploitation:

You start with a broad view of the data, you experiment, you work out the items you need and then you work on trimming down the solution so that the outcome is blisteringly fast. This way, we make sure that when we go to production, we have an efficient application that can scale to many, many users, and we provide each user a pleasant experience with a responsive offering.

I’ll spare you the modifications I made to my database, the elaboration of the schema and the level metrics – thanks to the initial exploration via the data import and the test dossier, I knew what results to expect. This saved me quite a bit of time.

I ended up with a schema cube that builds everything I need in a matter of minutes:

Menu

And a test report that validated the level metrics and the aggregations:

Schema Cube

As you can see in the above screenshot, I now have an attribute for the LSOA groups allowing me to select aggregate values for the towns I want to investigate. I also have the level and compound metrics that give me, by crime category, the percentage for the outcomes.

In passing, I notice that only a tiny proportion of reported crime results in the perpetrator being punished. This varies by crime type, but not by much. This needs investigating !

I’m now ready to start the second phase of the exploration, able to slice and dice with a much sharpened knife. And this is a good point to conclude this first part. Time permitting, I shall write the second part in the next few weeks.

Thanks for reading this far ! And here’s a teaser from my freshly minted cube:

mk crime type

And if you’re lucky, I might even make a video.