In recent years, both new technologies and new business practices have changed what the Database Administrator (DBA) role means, and started to shape how the role may evolve, or recede in the future. These changes offer exciting opportunities for DBAs, and the future looks great for data professionals overall. This post discusses the factors influencing the DBA role, how they may be affecting DBA numbers, and suggests some positive countermoves that DBAs might choose to make in response.
Advances in the capabilities and popularity of “fully-managed” database-as-a-service (DBaaS) offerings, like Azure SQL Database, tempt companies to migrate their relational databases from being hosted on-premise, to a cloud-based solution. As this happens, a number of the database maintenance tasks that would have traditionally been carried out by a DBA, can now be handled automatically (to varying extents) by the managed instance. These can include such fundamental DBA tasks as backups, high-availability, performance tuning and deciding whether indexes should be added or removed.
As well as managed databases, new business practices have also had an impact upon tasks that historically formed part of the DBA workload. The DevOps methodology brings developers together with operations staff, to collaborate on the whole application life cycle, through design, development, review, release and production support. This is an alternative to the traditional “sysadmin” model of developers and operations teams being “siloed” as separate entities (with the DBA firmly ensconced on the Ops side). Developers would write code that was then “thrown over the wall” for the Ops side to support. In a DevOps model it will make sense to “shift database changes to the left” and streamline whatever DBA-tasks the widely-skilled DevOps team think that they can automate.
Google’s Site Reliability Engineering (SRE) practice builds upon DevOps, but goes even further to rearrange the “Ops” DBA-tasks. Google explain that “SRE is fundamentally doing work that has historically been done by an operations team, but using engineers with software expertise, (to) design and implement automation with software to replace human labor”. I witnessed this first hand in my last permanent role, when my company introduced their take on SRE. New applications would now be developed, maintained and monitored by product-aligned engineering teams. Their responsibility would include supporting the database layer of the application, which would have previously been looked after by my DBA team.
What this means for the DBA
So, with the rise of managed DBaaS offerings and DevOps / SRE teams taking on database management themselves, it might be reasonable to think that there would be a reduction in the need for dedicated Database Administrators in the industry. This theory seems to backed up by looking at the number of DBA role vacancies in recent years. The graph below shows the total number (permanent + contract) of vacancies in the UK, recorded by IT Jobs Watch in the six months to January of each year, with Database Administrator opportunities falling substantially since 2017.
In their recent “The Future of Jobs Report 2018”, the World Economic Forum (WEF) find that operations-style roles are indeed falling, and specifically mention DBAs at one point: “historic hiring trends reveal a decline in hires of technical professions, such as Database Administrators and Electrical Engineers in South Asia”
What can DBAs do to adapt to this changing climate?
With the traditional Database Administrator position seemingly in decline, what can DBAs do to prolong and evolve their careers as data professionals?
The way I see it, there are three main options:
Carry on being a DBA (and get really good at it!)
As the “job vacancies” graph shows, there are still a large number of DBA roles appearing on the market, even if these are diminishing.
Plenty of organisations are still going to be using on-premise relational databases, and have a need for a DBA for many years to come – especially those large companies with vast databases, supporting numerous applications. Anyone who has worked in such organisations will know how difficult it can be just to upgrade the database environment to a newer version, or move it to different physical hardware – both from a practical point of view, and often even more so because of risk adversity from decision makers.
Moving to the cloud will be an intimidating prospect, and some companies will no doubt put stability and an unwillingness to risk disrupting customers, over the potential for cost-saving and access to new features. Some businesses may choose an on-premise database infrastructure due to the degree of control it offers (in certain industries, there may even be a legal obligation to keep customer data in a particular location). New business practices like DevOps / SRE won’t be adopted by all, with some organisations preferring to keep their data under the guardianship of a dedicated DBA.
However, it would be dangerous to rely upon reluctance to change within some organisations to assume the demand for DBAs will continue to match the supply. It is clear that a growing number of traditional DBA tasks are now being handled by cloud-based DBaaS offerings and widely-skilled engineering teams, so there is likely to be more of a fight to hang on to one of the declining number of DBA roles available.
To assist you in this fight, it could be worthwhile to get as good as you can be at the role. Keep learning and practising new skills, consider blogging or speaking at events to show what you know, and maybe look at getting some qualifications to distinguish you from the competition (I’ve just taken a couple of exams to gain the latest Microsoft Solutions Associate in Database Administration qualification).
The passing of DBA tasks to managed-databases or other teams doesn’t have to be bad thing for the DBA, but an opportunity for you to learn more, and focus upon higher-value activities, while the more repetitive and mundane tasks are automated-off of your daily workload.
In their recent “Data Never Sleeps” report, DOMO stated that “By 2020, it’s estimated that for every person on Earth, 1.7MB of data will be created every second… and we have a feeling things are just getting started”. More data is being created than ever before, and is playing a bigger part within business and society as a whole. All of this data needs to stored (even if perhaps not in on-premise relational databases), managed, and worked with.
As the “job vacancies” graph above shows, the number of Data Engineer roles, and positions requiring someone with “Azure SQL Database” skills have increased in recent years. The aforementioned “The Future of Jobs Report 2018” makes the positive overall estimate that “75 million jobs may be displaced by a shift in the division of labor between humans and machines, while 133 million new roles may emerge”. Although the WEF predict a decline in hires of some technical professions, they list both “Data Analysts and Scientists” and “Database and Network Professionals” in their top 10 emerging job roles within the UK. That top 10 is across all industries, not just IT and technology, so overall the forecast is really encouraging for data professionals as a whole.
Someone with a DBA skill set will be at a great starting point to build upon these skills and diversify slightly, to find a place in this new world of data.
With their knowledge of database design, different data types, and how to extract data from a source efficiently, a DBA will be well-placed to transfer these skills to more of a data-engineering role, moving and combining data from various sources and presenting them in an easily-digestible form for analysts to work with. A DBA with years of practice securing databases will be ideally positioned to apply these principles to help secure wider cloud environments. And even in fully cloud-based managed-database settings there is still going to be a need for someone to design the environment in the first place, choose the right data platform, suitable storage and redundancy options, and migrate on-premise databases to the cloud. A DBA who has had the chance to carry out the more architectural aspects of designing a new database environment should be an ideal candidate to fulfil this role.
You could take data as a starting point and diversify into other areas that interest you, widening your role to be able to do more with data, that will benefit more people within your organisation. If this sounds intimidating, remember that you’re a DBA, and being a DBA is HARD! Understanding query plans and wait stats is complicated stuff, and if you’ve successfully mastered this then you can basically learn anything!
Become a DBA-as-a-service
Organisations using managed database solutions, and companies that have dispersed a lot of their traditional DBA tasks between new engineering teams, will still have an occasional need for expert DBA advice. For example, managed databases can automatically add and remove indexes and adapt query plans, but applications using them will still benefit from a database expert advising on good query design. Backups may be performed automatically now – but are organisations going to be so happy to automate restores too, or will there still be a need for someone to make that call of when to restore? A dedicated DBA will continue to be able to add value within these models, on particular pieces of work. There just may not be a need for this DBA service all of the time, or as often as there was previously.
Just as DBaaS products can have their resources scaled up or down when required in a pay-per-use model, there is likely to be a desire for DBA expertise to be scaled up and down as required too. It may make sense for some DBAs to align themselves with the pay-as-you-go model by offering their DBA expertise to those who require it, on an adhoc basis, in effect becoming a DBA-as-a-service (DBAaaS).
An obvious way to make this work would be looking to enter the contracting market, and responding to the calls of organisations who need a boost of DBA expertise on a short-term basis. A less obvious alternative may be finding a role at a Managed Service Provider, who offer themselves as a DBA resource (usually amongst other skills) to a wide range of companies, as and when they need that resource. Information Age quote the “2017 IDC Application Services Survey” as saying “30% of executives are outsourcing management of their organisation’s infrastructure and application portfolio – this is expected to rise to 41% over the next five years.” This growth makes sense, if companies increasingly have a demand for operational support on an adhoc pay-per-use basis. It should also follow that requirements for DBA expertise within these Managed Service Providers grow alongside their popularity, presenting a greater number of opportunities for DBAs to work for such organisations.
While the prospect of supporting a wide range of companies in a plethora of industries, working with a greater range of database environments, and helping out a constantly changing cast of characters will excite some DBAs, this way of working may not be for everyone. However, it is possible for the DBAaaS concept to work within a single organisation too. If a company moves towards using a DevOps / SRE methodology, this will result in a greater number of engineers from a non-DBA background becoming custodians of their own data, which will be a new experience for them. They will need some guidance, and a DBA could act as a consultant or coach to the engineers – instilling in them DBA principles of security, monitoring and restore strategies, and embedding themselves within a particular engineering team when that team need to carry out a complex piece of database work as part of a project.
With the growth of managed database solutions and alternative working practices, the roles and responsibilities of the traditional Database Administrator are changing, and will continue to do so in the future. This by no means has to be a bad thing for the DBA.
There will be opportunities for us to realign ourselves alongside these new working methodologies and technologies by becoming a DBA-as-a-service, and providing an essential source of expertise that will enable the new arrangements to succeed.
Personally, I can’t think of anything more rewarding than empowering others with new skills. If you’re approached to help hand-off some of your DBA tasks to a person or application, embrace it and help make it work. Hoarding knowledge to try to prolong your current role will only make you look and feel bad. Furthermore, it’s very unlikely to work. If someone has already identified that part of your workload can be automated, or given to another team, then they won’t just give up on that idea if you make it hard to implement.
Grasp the opportunity that automation and knowledge-sharing present, and use the time it saves you to dedicate yourself to more high-value tasks. Spend some of this saved time equipping yourself with new skills that will enable you to diversify into one of the many emerging data-based professions, if and when required. Data will continue to be become increasingly central to business and society, and DBAs are excellently positioned to be an integral part of that world. Good luck!