Chapter 4

Roles in Big Data Revealed

In This Chapter

arrow Understanding different big data roles

arrow Examining real-world examples of requirements

arrow Seeing how job requirements translate to work life

arrow Doing a skills and interest assessment for each role

They say that when someone is training to spot counterfeit currency, the would-be crime fighter examines the real thing with more intensity than the fraud. That’s where examining real-world big data case studies comes in handy. In this chapter, I examine both the theory and the practical knowledge to help you craft your interview story and land that perfect job. I give you a look at different roles in big data along with real-life job posting case studies and interest assessments that help you gauge your interest in a particular big data field.

remember.eps Big data is a tool. There are many dimensions to the roles available in big data. In this chapter, we’ll build a foundation of different roles from a business and technical perspective.

Big Data Jobs for Business Analysts

Big data projects originate with solving problems with some business objective in mind. Much of the focus today centers around technology implementation, visualization tools, and data products, but it’s important to remember that technology with no end in mind has little business value. Enter the role of the business analyst. Some people claim that this career is an endangered species, but there is some very good news for business analysts. Big data isn’t just a new technology. It’s changing the face of how we do business, and that means that the business analyst’s role in big data is extremely important. It has been expanded to include that of business architect.

The basis for any of the roles discussed in this chapter often comes from the vision cast by business analysts. If you can envision a bridge that spans the gap between business and technology, you may find great success in this type of role. A business analyst can serve within a corporate IT division, a software firm specializing in big data, or a consulting firm. (See Chapters 8 and 10 for more information on life within these types of organizations.)

Some more good news, by the way: A recent Robert Half salary report shows the average salary for a business analyst is between $75,000 and $109,000, up more than 4 percent from 2013. Business intelligence analysts are seeing an even greater increase in starting salaries from 2013, with an increase of more than 7 percent. The market is demanding more analysts, and it’s paying for it.

technicalstuff.eps Besides the official title of business analyst, other possible job titles include marketing analyst, data analyst, and system analyst. (The term data analyst can also be confused with the data scientist’s role, as I explore later in this chapter.)

Assessing your interest

In this section, I fill you in on some attributes you should consider as you evaluate your skills and interests. Spend some time reflecting on these areas. Do the skills self-assessment in Chapter 2. Talk to trusted advisors and get their perspective on you. Look back at your reviews from previous jobs or class reviews if you’re still a student.

If you answer “yes” to many of the following questions, the business analyst role could be for you. Keep in mind, this is not an all-or-nothing guide. If you answer “no” or “not really” to a question, that doesn’t mean you should rule out a role as a business analyst.

Are you naturally inquisitive?

The best approach to big data analytics is to come at business problems with the question/hypothesis perspective. Business analysts need the industry expertise (or ability to collaborate with industry experts) to identify the most relevant and most valuable questions to explore.

Can you see beyond the surface issues and go deeper into the problem? Do you know when a good idea has potential? Business analysts are skilled at sticking with a problem until they’ve found a solution. If you can drive hard and get to an answer, this could be great role for you.

Can you see through to the end quickly?

A friend of mine who is a lieutenant commander in the U.S. Navy often says he looks to develop an important trait in junior officers. He’ll tell them, “Know the right answer when you hear it.” In other words, do you know when you’ve uncovered the right area to focus on, and do you pivot quickly to focus your energies on solving that problem?

One of the biggest challenges in big data is that there is way too much data — not too little. Business analysts who can quickly see what is just a distraction and what needs focus are very effective.

Can you shift between creative and analytical?

I sometimes think of big data analysis in terms of an alternate blend of left-brain and right-brain activities. Creativity, curiosity, and imagination are all needed, as well as logic and rational and critical thinking. This is perhaps the rarest attribute. People tend to have a bias toward either creativity or logic, but the well-balanced analyst has the ability to see things at a abstract level and then to quickly go deep into the issue. Can you build a presentation for an executive to explain an idea and then write a four-page detailed document to explain the economics, technology, or implementation strategy? If so, you might make a successful business analyst.

Do you understand your audience?

One of the biggest opportunity areas I see right now is the improvement of how information is communicated to decision makers. Business analysts who can convert data into business opportunities and recommend action will rise to the top. There is absolutely no business value in data unless it translates to action.

Can you talk technology with the CTO and also explain the financial benefits of big data to the CFO? Can you help the marketing manager see the impact to her business unit? A good big data business analyst doesn’t just understand big data technology and how it works; he also understands the impact to business and can speak the language of business.

remember.eps Business analysts need to have people skills, as well as communication skills. They need to like to work collaboratively and make presentations on and off the white board. They also need to write, document, and negotiate.

Looking at a job posting

The job postings for business analysts vary based on the type of company — whether it’s a consulting firm, a big data software firm, or an internal big data team for a corporation. These postings tend to be less specific in responsibilities and focus on solving business problems, good communication skills, and a balance of analytical ability and technology. You often see requirements for familiarity with Microsoft Excel, analytics tools, and database technologies. Largely though, the analytical skills are focused on problem-solving frameworks rather than database programming. A problem-solving framework follows a pattern for solving problems and executing on the solution. You need to be able to quickly identify the problem or need, find a solution, make recommendations, identify risks and how to avoid them, and describe what the action plan should be.

Consider the following job posting for an analyst with a big data focus. Carnegie Mellon University has published samples of business intelligence roles that recent employers have used. The following posting is for a business intelligence analyst, taken from Carnegie Mellon.

  • Business Intelligence Analyst
  • Each Business Intelligence Analyst is aligned with one or more groups, such as marketing, logistics, or customer service, and partners with those teams to help them achieve their goals. Whether you’re measuring site performance, analyzing customer behavior and trends, data mining, or optimizing SQL queries, you’ll be working with cutting-edge technology and multi-terabyte datasets. Working on the Business Intelligence team is a premier opportunity to develop a career in business and big data analytics.
  • At their core, Business Intelligence Analysts are strong in quantitative analysis. They enjoy coding but also want to balance that with their interest in business. They think critically to tackle complex challenges, thrive in a fast-paced environment, and are seeking a high-growth opportunity where they’ll have an immediate impact on day one. Business Analysts are strong communicators who are eager to learn, are endlessly curious, take pride in hard work, and are committed to rapidly advancing their career.

    Responsibilities include:

    • Consulting with internal customers (for example, marketing, logistics, or customer service) to develop analyses that lead to actionable insights that accelerate profitable growth
    • Wrangling data from multiple sources including sales, inventory, product, and customer databases to create integrated views that can be used to drive decision making
    • Working with several large and complex SQL databases
    • Designing and building reports and analyses in Microsoft Excel
  • Qualifications include:
    • Highly analytical data junkie who enjoys coding but doesn’t want to be a software engineer
    • Positive, people-oriented, and has an energetic attitude
    • Analytical, creative, and employs an innovative approach to solving problems
    • Strong written and verbal communication skills
    • Entrepreneurial spirit
    • Degrees represented on current Business Analyst team include: Economics, Computer Science, Engineering, Physics, and Music

There are a few things worth calling out in this posting that can help you decide if this role is for you. In the list of responsibilities, the positing says, “Consulting with internal customers (for example, marketing, logistics, or customer service) to develop analyses that lead to actionable insights that accelerate profitable growth.” What does that mean really? Analysts don’t just have to understand information; they need to be able to articulate an action plan so that the business can capitalize on those insights. This is not merely a role that notices interesting things. This individual is expected to draw conclusions and drive action to revenue.

This role is technical, but you aren’t expected to do heavy programming. Should you be able to code? Yes, but you probably won’t be doing much of that. That’s important for those who build those virtual bridges between business and technology — they need to be able to understand the components of big data solutions like appropriate technologies, software, or hardware needed to fulfill the business requirements. If the technology team has selected one programming language or model over another, the business analyst needs to be able to understand why that’s a good or bad decision and how that could impact the overall outcome.

Finally, check out the kinds of majors that fall into this role — pretty much everything. Employers are looking for problem solvers who can find creative solutions and have the bias for action to drive real results.

Big Data Jobs for Data Scientists

Data scientists take the recommendations that the business analysts make and do a variety of tasks including the following:

  • Build the technical case. They apply advanced math and statics to build the technical cases around the hypotheses that the business analysts build. Data scientists are tasked with building the models required to test these theories. This model is important to big data. You start with a hypothesis. For example, if we change the branding colors on a product on a given day and publish that on Twitter and it is positively received, we can expect an increase in sales of 4 percent. That is the hypothesis.
  • Create the mathematical models. These models measure what positive sentiment means and then can model what tests need to be run to find correlations between that and price increases.
  • Discover patterns, trends, and correlations. Some tasks may not necessarily start with a hypothesis. This is where the real power of big data comes in. You find patterns and trends you didn’t even know existed.

The skill required here is to take a business idea and model it with numbers and data. Data scientists take that data and turn it into information. There can be a fine line between what data scientists do and what computer scientists do. There are some overlaps, but there are also jobs with a significant difference, namely in scientific and academic research.

Assessing your interest

As with the business analysts, there are a set of questions you can ask yourself to see if you’re a fit for this type of job. Roles as a pure data scientist often require a master’s degree or a PhD. So, you should carefully consider the following questions.

Are you naturally inquisitive?

Just as a business analyst needs to think in terms of building hypotheses, the data scientist needs to have aptitude in this area. Computer scientists need to be able to construct models that can prove or disprove a given business hypothesis. Can you see beyond the surface issues and go deep? Do you know when a result has potential and needs further testing? Are you passionate about technology?

Can you focus for a long time?

The journey required to complete a PhD or advanced degree in the big data field (see Chapter 5) can be a long one. You have to commit a significant amount of study to a specific area of research. Are there areas of math, statistics, or computer science that you have a passion for studying? Do you want to address big problems that may take years to solve? Do you like to write … a lot? Can you maintain intense focus on a few topics for many years — maybe for an entire career?

Are you self-motivated?

Data scientists need to be able to direct their own intellectual paths. Do you naturally follow a solution to its end? Do you have a knack for knowing where to find answers if you don’t know them?

Are you multidisciplined?

Data scientists need to be knowledgeable in multiple areas — math, statistics, and computer science. Can you pick up computer science languages and concepts easily? Does the idea of a new language excite you or intimidate you? Can you easily collaborate with others to learn new things?

Idea to reality

Data modeling requires the ability to take business concepts and ideas and model those within a world driven by numbers and data concepts. Do you have the aptitude or interest to build experiments that capture the business value?

Looking at a job posting

Let’s take a look at job posting for a data scientist who would operate at a junior level, or someone who has less than five years experience. The first posting is for an entry-level consultant, and the second would be more aligned with an academic or research-oriented position and was actually posted on several job search websites such as Indeed and SimplyHired. Both are grounded in math and statistics.

  • Data Consultant — Recent College Grad
  • Are you a recent college graduate who loves big data? Are you passionate about cutting-edge technologies and solving challenges for Fortune 500 clients? As a consultant, you’ll be part of a team that develops and implements advanced algorithms and data pipelines that extract, classify, merge, and deliver new insights and business value out of structured and unstructured data sets. You’ll work on a team whose data science efforts range from exploration and investigation to design and development of analytic systems. You’ll have a chance to gain diverse experience across multiple technologies and create path-breaking solutions. You’ll be surrounded and learn from the foremost Thought Leaders in the big data space.
  • This posting describes two paths: Data engineering and data science.
  • Key responsibilities include:
  • Data engineering
    • Designing and developing code, scripts, and data pipelines that leverage structured and unstructured data integrated from multiple sources
    • Software installation and configuration
    • Participating in requirements and design workshops with our clients
    • Developing project deliverable documentation
  • Data science
    • Providing big data solutions for our clients, including analytical consulting, statistical modeling, and quantitative solutions
    • Mentoring sophisticated organizations on large-scale data and analytics and working closely with client teams to deliver results
    • Helping to translate business cases to clear research projects, be they exploratory or confirmatory, to help our clients utilize data to drive their businesses
    • Collaborating and communicating across geographically distributed teams and with external clients
  • Required skills/experience include:
  • Data engineering
    • BS or MS in Computer Science or equivalent work experience
    • Experience programming in Java, Python, SQL, or C/C++
    • Background that includes mathematics, statistics, machine learning, and data mining
    • Experience with SQL, NoSQL, relational database design, and methods for efficiently retrieving data
    • Prior work/research experience with unstructured data and data modeling
    • Strong analytical skills and creative problem solver
    • Excellent verbal and written communications skills
    • Strong team player capable of working in a demanding startup environment
    • Experience building complex and noninteractive systems (batch, distributed, and so on)
  • Data science
    • BS or MS in Computer Science, Math, or equivalent work experience
    • Coursework in mathematics, statistics, machine learning, and data mining
    • Proficiency in R or other math packages (Matlab, SAS, and so on)
    • Excellent programming skills in object-oriented languages
    • Adept at learning and applying new technologies
    • Excellent verbal and written communication skills
    • Strong team player capable of working in a demanding startup environment
    • Experience with Java and Python
  • Research Scientist Physiological Data Modeling
  • The Henry M. Jackson Foundation (HJF) is looking for junior and senior scientists to join the U.S. Army Medical Research and Materiel Command’s Biotechnology High Performance Computing Software Applications Institute (BHSAI; www.bhsai.org). HJF provides scientific, technical, and programmatic support services to the BHSAI.
  • This opening is for dynamic scientists who are interested in
    • Working with colleagues in other disciplines
    • Applying computational solutions to biomedical problems
    • Doing signal processing of time series physiological data
    • Doing data mining, creating data-driven and physiological-based models, and working with artificial intelligence programs
  • The candidate should have a PhD in a related discipline and a strong publication record. The candidate is expected to simultaneously work on multiple projects, involving a diverse and interdisciplinary team of scientists across multiple laboratories.
  • Foreign nationals are welcome to apply. U.S. citizenship or permanent resident status is not required. This position is located in Frederick, Maryland.
  • The Henry M. Jackson Foundation for the Advancement of Military Medicine, Inc. (HJF) is a congressionally authorized, not-for-profit corporation that provides unparalleled scientific and management services to military medical research and education programs worldwide. Our mission is to advance military medical research. AA/EEO (Affirmative Action/Equal Opportunity Employer).

Two main things are important to point out in these postings:

  • You don’t have to have a PhD to be a data scientist. The first role of a data engineer requires the candidate to have deep understanding of data modeling, programming, machine learning, and math. Although they aren’t building complicated algorithms oriented around research like the second posting, this role requires a deep understanding of data and how to construct data to extract value.
  • There is some apparent crossover in function with the computer scientist’s role. The distinction is made around the matter of expertise the employee will have. The reference in the second role to having “a PhD in a related discipline and a strong publication record” means that the candidate has done real research in the area of big data and has published findings in an academic journal. This isn’t just being able to spin up Hadoop clusters or create complex data stores. Data scientists have the skill and interest in using big data to conduct research and development. This requires a special set of skills and a desire and ability to publish in scientific and scholarly journals. The most significant aspects of the postings are having a foundation in math and statistics, data mining, and mathematical modeling. These are the everyday tools of the data scientist.

Big Data Jobs for Software Developers

Big data projects originate from solving problems with some business objective in mind. Much of the focus today centers around technology implementation, visualization tools, and data products. Today, businesses are doing more with less and need to show the return on investment in everything they do. Software developers are tasked with translating the business problems into workable solutions that drive revenue to the bottom line. (See Chapters 8 and Chapter 9 for more information the life of data practitioners.)

Assessing your interest

Big data jobs for software developers require many of the same core interests as other software developer jobs, but with a twist. Software jobs in this world are not static. Things change a lot — like new technologies, associated languages, software frameworks, and programming techniques. If you love solving problems with code, that’s a good start.

Are you a team player?

This isn’t just a cute cliché. The days of getting an assignment, going into an office for a few weeks, and coding in the dark are long gone. The industry is quickly migrating to the world of agile development, which focuses on software outcomes through a very iterative and collaborative approach. Teams are typically very small and co-located. You need to be able to work well with all stakeholders, not just your boss or team members, but customers as well.

Do you know more than one coding language?

Most software developers have more than one coding language under their belts. For a big data developer, that’s just the start. Do you have the ability to learn and use new languages? Can you easily learn these languages on your own? Can you pivot between coding languages easily? In many big data software projects, developers need to be able to shift from using scripting languages like Python, to customizing a Hadoop job in Java, to turning a relational database data query. If you’re more comfortable in predictable, steady-state software development, you may not thrive as a big data developer.

Are you ready to learn?

Not only do you learn new languages all the time, but you learn new techniques and frameworks. Big data is advancing so rapidly that staying current is challenging. The exciting thing about this technical advancement is that it isn’t only around software — it’s in hardware and cloud services as well.

A good indication of your ability to learn about big data is to look at your thirst for it. Are you constantly reading and trying new things?

Looking at sample job postings

The job postings in this section are for a big data developer and data scientist/software engineer. Notice that both jobs require more experience. You’ll need to be able to demonstrate that the experience you have directly ties to the requirements. Think through all the duties you’ve had and see what skills have been developed as a result of that experience.

  • Big Data Developer
  • Support the creation of web-based decision support and analytic tools, using the latest JavaScript libraries. Maintain responsibility for the design, development, and sustainment of various existing and new web applications. Research and apply the latest web technologies to meet client requirements and develop rapid prototypes. Provide support to government and commercial clients, analyzing operational capabilities related to information technology functionality, integration, and interoperability issues. Perform research and analysis, support visualization design related to requirements traceability, portfolio management, and programmatic risk and health, and develop innovative scientific simulations and data visualizations.
  • Qualifications include:
  • Basic qualifications:
    • 5+ years of experience with object-oriented programming or entire software development life cycle in an academic or professional environment
    • Experience with algorithm theory and data structures
    • Ability to learn new programming languages and architectures quickly
    • Ability to obtain a security clearance
    • BS degree
  • Additional qualifications:
    • Experience with researching, presenting, briefing, or communicating analytical material
    • Experience with designing and implementing complex simulations
    • Experience with rich Internet application frameworks, including AJAX, Silverlight, Flex, or ActionScript
    • Experience with development of applications for scientific visualizations
    • Experience in a rapid prototyping environment
    • Excellent oral and written communication skills
    • BS degree in Computer Science
  • Big Data Scientist/Software Engineer
  • You’ll be working with a team of smart researchers and engineers to address challenging data problems in the exciting mobile advertising domain. The goal of the team is to make sense out of the huge amount of data flowing through our proprietary ad platform. The problems we work on include, but are not limited to, yield optimization, smart pricing, network behavior analysis and modeling, campaign performance optimization, inventory prediction, and management.
  • Responsibilities include:
    • Development, research, and exploration in the areas of statistics, machine learning, experimental design, and operational research
    • Propose, design, and analyze new algorithms to benefit our ad network
    • Research and design experiments to evaluate different algorithms’ impact on the network
    • Implement and verify new algorithms, and integrate the algorithms into production
    • Work with various other team members including data analysts, business owners, engineering, product management, and trafficking
  • Experience/skills include:
    • Master’s or PhD in engineering, data mining, statistics, operations research, math, physics, economics, or equivalent required.
    • Two to five years of software engineering and scientific experience.
    • Experienced with Java, Python, and big data.
    • Experience with Hadoop, Hive, and Pig a plus.
    • Prior ad network, big data, or analytics experience, operations research, or similar. Proficiency with databases, SQL, and scripting languages.
    • Experience in data mining, data matching, machine learning, statistical techniques, experimental design, or optimization. Experience in extracting and manipulating extremely large datasets.
    • Practical understanding of the mathematics behind modern machine learning, linear algebra, and statistics.
    • Demonstrated ability to apply statistical techniques to solve real problems. Experience with data analysis, business intelligence, and statistical tools (for example, R, SAS, or SPSS). Analytical thinker.
    • Ability to work independently and in a team to research innovative solutions to challenging business/technical problems.
    • Attention to detail, data accuracy, and quality of output.
    • Results oriented and deadline driven.