© The Author(s), under exclusive license to APress Media, LLC , part of Springer Nature 2021

B. E. HollisterCore Blender Developmenthttps://doi.org/10.1007/978-1-4842-6415-7_4

4. The blenlib and blenkernel Modules

Brad E. Hollister¹  

(1)

Computer Science Department, CSUDH, Carson, CA, USA

 

In this chapter, we discuss Blender’s two largest “core” utility libraries. blenlib and blenkernel are the primary modules for generic processing and data. As such, they share a history in the Blender codebase. When appropriate, we will use Blender’s unit tests—written with Google Test¹—to survey blenlib’s interface (BLI_* functions). Recall struct bContext is defined in blenkernel. We looked at the CTX_* API, a subset of the blenkernel API in Chapter 2. blenkernel’s API is generally accessed by BKE_* functions, which comprise a larger portion of its interface. blenkernel offloads data handling for makesdna, by operating on “DNA” data types. Other utility modules, namely, blenloader, blenfont, and blentranslation, are more specialized and beyond the scope of the current chapter.

Overview of blenlib

The blenlib module is quoted by the official documentation as:

Internal misc libraries: math functions, lists, random, noise, memory pools, file operations (platform agnostic)

—https://wiki.blender.org/wiki/Source/File_Structure

blenlib’s dependencies are shown in Figure 4-1. Many other modules are themselves dependent on blenlib (Figure 4-2).

Figure 4-1

blenlib’s dependencies. It has limited dependency on other modules. There is some reliance on the repository’s internal support libraries, shown in the red

Figure 4-2

Partial dependency graph for both blenlib and blenkernel. Note that blenlib, blenlib, and makesdna each have a relatively high number of dependent modules

Google Test

One approach to exploring blenlib is to read the unit tests from source/blender/blenlib/tests/. Almost every source file in source/blender/blenlib/intern/ has a corresponding test file having .cc extension.

First, we will need to briefly describe Google Test and to a lesser extent CMake’s CTest facility. The Blender source tree contains unit tests for functions at the module API level. The number of modules tested, however, is limited. Blender uses the gtest (Google Test) for unit test implementation. Test cases are class derivations from gtest’s Test class. gtest provides a macro to simplify writing tests (Listing 4-1).

TEST(TestSuiteName, TestName) {

  ... test body ...

}

Listing 4-1

TEST() is provided by gtest. The format of Blender unit tests is shown

Running Tests

blenlib unit tests are compiled with the WITH_GTESTS option enabled in source/blender/blenlibblenlib/CMakeLists.txt. They are added to the platform-dependent build scripts produced by running the CMake program. A target is added to the build script, so that when it is run as part of a system build, the unit tests can be executed automatically by the build system (i.e., Make, Visual Studio IDE, etc.). A build script calls ctest, an executable provided by the CMake distribution. ctest runs tests registered by CMakeLists.txt .

...

if(WITH_GTESTS)

  set(TEST_SRC

    tests/BLI_array_store_test.cc

    tests/BLI_array_test.cc

    tests/BLI_array_utils_test.cc

    tests/BLI_delaunay_2d_test.cc

...

Listing 4-2

Partial listing for source/blender/blenlibblenlib/CMakeLists.txt. Note that the tests are compiled into a single executable blenlib_tests. This executable is written to the build directory in bin/tests/

It is not necessary to use ctest, in order to run unit tests. Unit tests are compiled to the build directory, in bin/tests/. All tests are compiled into blenlib_tests.

blenlib’s Unit Tests

In addition to gtests macros for assertions, Blender adds some for data types defined in blendlib (Listing 4-3). An example is EXPECT_V3_NEAR(), used for three-dimensional vector data types, defined in terms of EXPECT_NEAR()—a gtest macro .

#ifndef __BLENDER_TESTING_H__

#define __BLENDER_TESTING_H__

#include <vector>

#include "gflags/gflags.h"

#include "glog/logging.h"

#include "gtest/gtest.h"

namespace blender::tests {

/* These strings are passed on the CLI with the --test-asset-dir and --test-release-dir arguments.

 * The arguments are added automatically when invoking tests via `ctest`. */

const std::string &flags_test_asset_dir();   /* ../lib/tests in the SVN directory. */

const std::string &flags_test_release_dir(); /* bin/{blender version} in the build directory. */

}  // namespace blender::tests

#define EXPECT_V3_NEAR(a, b, eps) \

  { \

    EXPECT_NEAR(a[0], b[0], eps); \

    EXPECT_NEAR(a[1], b[1], eps); \

    EXPECT_NEAR(a[2], b[2], eps); \

  } \

  (void)0

#define EXPECT_V4_NEAR(a, b, eps) \

  { \

    EXPECT_NEAR(a[0], b[0], eps); \

    EXPECT_NEAR(a[1], b[1], eps); \

    EXPECT_NEAR(a[2], b[2], eps); \

    EXPECT_NEAR(a[3], b[3], eps); \

  } \

  (void)0

...

Listing 4-3

Excerpt from tests/gtests/testing/testing.h. EXPECT_NEAR() is used for testing vector data

As an example of a unit test from BLI_array_test.cc (Listing 4-4), we see a test of Array, using the TEST() macro.

#include "BLI_array.hh"

#include "BLI_exception_safety_test_utils.hh"

#include "BLI_strict_flags.h"

#include "BLI_vector.hh"

#include "testing/testing.h"

namespace blender::tests {

TEST(array, DefaultConstructor)

{

  Array<int> array;

  EXPECT_EQ(array.size(), 0);

  EXPECT_TRUE(array.is_empty());

}

TEST(array, SizeConstructor)

{

  Array<int> array(5);

  EXPECT_EQ(array.size(), 5);

  EXPECT_FALSE(array.is_empty());

}

...

Listing 4-4

Partial unit test defined in source/blender/blenlib/tests/tests/BLI_array_test.cc. blenlib’s Array template class is tested

blenlib’s API

The unit test files for blenlib are shown in Figure 4-3. We see that each unit test file roughly corresponds to blenlib’s API headers. The file listing in Figure 4-4 is considerably larger, suggesting that the blenlib API is not fully tested. Notable omissions: BLI_convexhull_2D.h, BLI_allocator.h, and BLI_timer.h.

Figure 4-3

The blenlib (BLI_*) unit tests. Each file roughly corresponds to the interface files shown in Figure 4-4

Figure 4-4

blenlib’s interface headers. These files prototype BLI_* functions

Note

While it is virtually impossible to list the entire BLI_* API here, a good way to access documentation separate from the source code is by using Doxygen-generated hyperlinked documents. Doxygen produces pdf or html from JavaDoc-style comments. However, most of the blenlib module’s API is not commented, and in many instances its API is either inlined (with a definition in the header only) or simply a preprocessor macro. Most source files in source/blender/blenlib/intern/ contain statically linked function definitions (e.g., bli_* functions or lower-level helper functions). These are called only indirectly by blenlib’s API. The repository provides a Doxygen configuration (doc/doxygen/Doxyfile), ready to run on the source. The configuration may be used at the command prompt via doxygen Doxyfile. Doxygen must first be installed.

Features of the blenlib API

Parts of blenlib are implemented in C++.² Areas of blenlib’s concerns are

• Data structures

  • Linked lists

  • Maps

  • Sets

  • kdtree

  • Bounding-volume hierarchy (bvh)

• Graph algorithms

  • A* search

• Memory allocation

• Hashing

  • Open addressing

  • md5

  • mm2

  • mm3

• Dynamic strings

• Assertions

• Byte swapping for big- and little-endian conversions

• Command-line argument processing

• File operations, such as renaming files

• Path processing for files

• Sorting

• Thread management

• Timers

• Two-dimensional processing

  • Geometry, that is, “rectangles”

  • Noise generation (texture generation)

  • Polygon fill

  • Voronoi

  • UV projection

• Operating system–specific data types

• Bitmaps

• Console output

• Random number generation

• Vector and quaternion math utility functions

  • Higher-order uses of basic functions for “easing” (animation)

  • Convex hull (geometry) processing

  • Color

  • Matrix operations

  • Statistics

  • Rotation

  • Interpolation

  • Quadrics

• Voxels

blenlib implements byte operations (e.g., big- and little-endian conversion, etc.) used for blend files (see Chapter 2). As a reminder, blend files are essentially binary serializations of Blender “DNA.”

There are a few surprises. For instance, one is blenlib’s dynamic string type struct DynStr, defined in BLI_dynstr.c. (We have already seen that there is an internal library called string.) Another is that BLI_listbase.h has a dependency on DNA_listbase.h, where the functions used to insert and remove from Listbase are prototyped.

Note

blenlib contains some redundancy. For linked-lists there are BLI_linklist.h and its associated BLI_link_utils.h, vs. BLI_listbase.h and BLI_listbase_wrapper.h. Both provide very similar utilities for Listbase. These are essentially two separate linked-list implementations.

blenlib API Examples

Now for some API examples. We use Doxygen-derived call graphs to see where calls are made to the API. We also look at API function definitions.

equals_m4m4(), our first example, has its prototype in source/blender/blenlib/BLI_math_matrix.h and definition in source/blender/blenlib/intern/math_matrix.c. There are no comments in the codebase for equals_m4m4(). equals_m4m4()’s implementation is shown in Listing 4-5.

bool equals_m4m4(const float mat1[4][4], const float mat2[4][4])

{

          return (equals_v4v4(mat1[0], mat2[0]) && equals_v4v4(mat1[1], mat2[1]) &&

          equals_v4v4(mat1[2], mat2[2]) && equals_v4v4(mat1[3], mat2[3]));

}

Listing 4-5

equals_m4m4() takes two-dimensional arrays of const floats. The function compares each column using equals_v4v4(). If all corresponding column vectors of two 4x4 matrices have identical entries, then matrices are considered equal. A boolean value is returned

equals_m4m4() calls the vector utility function equals_v4v4(), as shown in Listing 4-6. We can see that equals_m4m4() calls equals_v4v4(). Only three functions from the codebase reference equals_m4m4(), shown in Figure 4-5. We see later that the Python interface provides access to equals_m4m4(). Therefore, it is used more frequently than suggested.

MINLINE bool equals_v4v4(const float v1[4], const float v2[4])

{

   return ((v1[0] == v2[0]) && (v1[1] == v2[1]) && (v1[2] == v2[2]) && (v1[3] == v2[3]));

}

Listing 4-6

The equals_v4v4() function . equals_v4v4() is inlined. This is specified by MINLINE, defined as static inline

Figure 4-5

Doxygen documentation for equals_m4m4()

Next we have the BLI_stack.h API. First, look at struct BLI_Stack, defined in source/blender/blenlib/intern/stack.c. BLI_Stack is shown in Figure 4-6 and Listing 4-7.

Figure 4-6

BLI_Stack is an aggregate of StackChunk objects. The open diamond denotes this as an aggregation, not composition. We may have a BLI_Stack object without the need for any StackChunk objects composing the BLI_Stack, that is, the stack can be empty

#define USE_TOTELEM

#define CHUNK_EMPTY ((size_t)-1)

/* target chunks size: 64kb */

#define CHUNK_SIZE_DEFAULT (1 << 16)

/* ensure we get at least this many elems per chunk */

#define CHUNK_ELEM_MIN 32

struct StackChunk {

  struct StackChunk *next;

  char data[0];

};

struct BLI_Stack {

  struct StackChunk *chunk_curr; /* currently active chunk */

  struct StackChunk *chunk_free; /* free chunks */

  size_t chunk_index;            /* index into 'chunk_curr' */

  size_t chunk_elem_max;         /* number of elements per chunk */

  size_t elem_size;

#ifdef USE_TOTELEM

  size_t totelem;

#endif

};

Listing 4-7

The BLI_Stack and StackChunk structs from source/blender/blenlib/intern/stack.c. CHUNK_EMPTY, CHUNK_SIZE_DEFAULT, and CHUNK_ELEM_MIN definitions are shown. In the CHUNK_EMPTY macro, the negative one is cast to size_t (unsigned int). The size_t data type is returned from the sizeof()

The interface for BLI_Stack operations is found in BLI_stack.h (Listing 4-8).

...

typedef struct BLI_Stack BLI_Stack;

BLI_Stack *BLI_stack_new_ex(const size_t elem_size,

                            const char *description,

                            const size_t chunk_size) ATTR_WARN_UNUSED_RESULT ATTR_NONNULL();

BLI_Stack *BLI_stack_new(const size_t elem_size, const char *description) ATTR_WARN_UNUSED_RESULT

    ATTR_NONNULL();

void BLI_stack_free(BLI_Stack *stack) ATTR_NONNULL();

void *BLI_stack_push_r(BLI_Stack *stack) ATTR_WARN_UNUSED_RESULT ATTR_NONNULL();

void BLI_stack_push(BLI_Stack *stack, const void *src) ATTR_NONNULL();

void BLI_stack_pop_n(BLI_Stack *stack, void *dst, unsigned int n) ATTR_NONNULL();

void BLI_stack_pop_n_reverse(BLI_Stack *stack, void *dst, unsigned int n) ATTR_NONNULL();

void BLI_stack_pop(BLI_Stack *stack, void *dst) ATTR_NONNULL();

void *BLI_stack_peek(BLI_Stack *stack) ATTR_WARN_UNUSED_RESULT ATTR_NONNULL();

void BLI_stack_discard(BLI_Stack *stack) ATTR_NONNULL();

void BLI_stack_clear(BLI_Stack *stack) ATTR_NONNULL();

size_t BLI_stack_count(const BLI_Stack *stack) ATTR_WARN_UNUSED_RESULT ATTR_NONNULL();

bool BLI_stack_is_empty(const BLI_Stack *stack) ATTR_WARN_UNUSED_RESULT ATTR_NONNULL();

...

Listing 4-8

Elided BLI_stack.h. Operation names are in boldface. ATTR_WARN_UNUSED_RESULT and ATTR_NONNULL() are defined in source/blender/blenlib/BLI_compiler_attrs.h. They specify attributes that can be verified at compile-time. For instance, with ATTR_NONNULL(), all arguments are checked by the compiler to be non-null. In the absence of any arguments to the macro, only pointers are checked to be non-null

BLI_stack_pop(), shown in Listing 4-9, has a number of references in the codebase (Figure 4-7). A bmesh operator function calls it from bm_face_split(), defined in source/blender/bmesh/operators/bmo_dissolve.c. The BLI_* API is used extensively throughout the codebase.

...

static void *stack_get_last_elem(BLI_Stack *stack)

{

  return ((char *)(stack)->chunk_curr->data) + ((stack)->elem_size * (stack)->chunk_index);

}

...

/**

 * Retrieves and removes the top element from the stack.

 * The value is copies to \a dst, which must be at least \a elem_size bytes.

 *

 * Does not reduce amount of allocated memory.

 */

void BLI_stack_pop(BLI_Stack *stack, void *dst)

{

  BLI_assert(BLI_stack_is_empty(stack) == false);

  memcpy(dst, stack_get_last_elem(stack), stack->elem_size);

  BLI_stack_discard(stack);

}

...

Listing 4-9

BLI_stack_pop() , from source/blender/blenlib/intern/stack.c. The code uses BLI_assert() and stack_get_last_element() for retrieving the top element; that is, a “chunk” of bytes, of elem_size and type struct StackChunk

Figure 4-7

Documentation for BLI_stack_pop(). BLI_stack_pop() is referenced by other BLI_* functions and from the broader codebase

Overview of blenkernel

Unfortunately, blenkernel currently does not have unit tests. We must review its directory and inspect its code directly. From official Blender documentation, the blenkernel module’s purpose is:

Kernel functions (data structure manipulation, allocation, free. No tools or UI stuff, very low level); kernel functions are shared with the blenderplayer, for loading data

—https://wiki.blender.org/wiki/Source/File_Structure

Consider the byte-swapping functions prototyped in BLI_endian_switch.h. It might be argued that these functions are low-level enough to reside in blenkernel. (blenkernel’s API headers are shown in Figure 4-8.) However, blenkernel’s data management routines are specific to Blender. For example, struct bContext (as indicated by BKE_context.h) is a Blender construct. Because blenkernel is tightly coupled with other parts of the Blender application codebase, it is difficult to describe the “kernel” functions without also dissecting the modules they assist.

Figure 4-8

Partial listing of the blenkernel module’s interface header files. These files prototype the BKE_* functions

Parts of the blenkernel module, such as those dealing with fluids (BKE_fluid.h, inter/fluid.c), character animation (BKE_anim.h, BKE_animsys.h, BKE_armature.h, intern/anim.c, intern/animsys.c, intern/armature.c), and other specialized areas, are peripheral to our exploration.

DNA Types and blenkernel

Most of makesdna’s data types have corresponding blenkernel API functions and associated files. These are listed in Table 4-1. blenkernel offers utilities for manipulating makesdna data types. Blender “DNA,” which is passed from one module to another, is operated on by functions from blenkernel.

Table 4-1

makesdna files and their blenkernel associated headers. Each BKE_*.h file contains API prototypes. These functions operate on corresponding data types defined in DNA_*.h.

makesdna File	blenkernel File
DNA_action_types.h	BKE_action.h
DNA_anim_types.h DNA_boid_types.h DNA_brush_types.h DNA_cachefile_types.h DNA_camera_types.h DNA_cloth_types.h DNA_collection_types.h DNA_color_types.h	BKE_anim.h BKE_boids.h BKE_brush.h BKE_cachefile.h BKE_camera.h BKE_cloth.h BKE_collection.h BKE_colortools.h
DNA_constraint_types.h DNA_curve_types.h DNA_curveprofile_types.h DNA_customdata_types.h DNA_dynamicpaint_types.h DNA_effect_types.h DNA_fluid_types.h DNA_freestyle_types.h DNA_gpencil_types.h DNA_image_types.h DNA_ipo_types.h DNA_key_types.h DNA_lattice_types.h DNA_layer_types.h DNA_light_types.h DNA_lightprobe_types.h DNA_linestyle_types.h DNA_mask_types.h DNA_material_types.h DNA_mesh_types.h DNA_modifier_types.h DNA_movieclip_types.h DNA_nla_types.h DNA_node_types.h DNA_object_types.h DNA_outliner_types.h	BKE_constraint.h BKE_curve.h BKE_curveprofile.h BKE_customdata.h BKE_dynamicpaint.h BKE_effect.h BKE_fluid.h BKE_freestyle.h BKE_gpencil.h BKE_image.h BKE_ipo.h BKE_key.h BKE_lattice.h BKE_layer.h BKE_light.h BKE_lightprobe.h BKE_linestyle.h BKE_mask.h BKE_material.h BKE_mesh.h BKE_modifier.h BKE_movieclip.h BKE_nla.h BKE_node.h BKE_object.h BKE_outliner_treehash.h
DNA_packedFile_types.h DNA_particle_types.h DNA_rigidbody_types.h DNA_scene_types.h DNA_screen_types.h DNA_sequence_types.h DNA_shader_fx_types.h DNA_sound_types.h DNA_speaker_types.h DNA_text_types.h DNA_texture_types.h DNA_tracking_types.h DNA_world_types.h	BKE_packedFile.h BKE_particle.h BKE_rigidbody.h BKE_scene.h BKE_screen.h BKE_sequencer.h BKE_shader_fx.h BKE_sound.h BKE_speaker.h BKE_text.h BKE_texture.h BKE_tracking.h BKE_world.h

blenkernel API Examples

BKE_world.h

The following examples are illustrative of blenkernel’s API for “DNA.”

The BKE_* module API functions prototyped in BKE_world.h are shown in Listing 4-10.

...

void BKE_world_free(struct World *sc);

void BKE_world_init(struct World *wrld);

struct World *BKE_world_add(struct Main *bmain, const char *name);

void BKE_world_copy_data(struct Main *bmain,

                         struct World *wrld_dst,

                         const struct World *wrld_src,

                         const int flag);

struct World *BKE_world_copy(struct Main *bmain, const struct World *wrld);

struct World *BKE_world_localize(struct World *wrld);

void BKE_world_make_local(struct Main *bmain, struct World *wrld, const bool lib_local);

void BKE_world_eval(struct Depsgraph *depsgraph, struct World *world);

...

Listing 4-10

Example BKE_world.h API prototypes. Function names are in boldface

BKE_camera.h

The BKE_* module API functions prototyped in BKE_camera.h are shown in Listing 4-11.

...

/* Camera Datablock */

void BKE_camera_init(struct Camera *cam);

void *BKE_camera_add(struct Main *bmain, const char *name);

void BKE_camera_copy_data(struct Main *bmain,

                          struct Camera *cam_dst,

                          const struct Camera *cam_src,

                          const int flag);

struct Camera *BKE_camera_copy(struct Main *bmain, const struct Camera *cam);

void BKE_camera_make_local(struct Main *bmain, struct Camera *cam, const bool lib_local);

void BKE_camera_free(struct Camera *ca);

/* Camera Usage */

float BKE_camera_object_dof_distance(struct Object *ob);

int BKE_camera_sensor_fit(int sensor_fit, float sizex, float sizey);

float BKE_camera_sensor_size(int sensor_fit, float sensor_x, float sensor_y);

...

Listing 4-11

Example BKE_camera.h API prototypes. Function names are in boldface

As we have discussed, blenkernel provides functions for operating on various Blender objects. Importantly, not all of these objects originate in makesdna. Two prominent examples, struct Main and struct bContext, are in fact defined in blenkernel itself. Note that struct Main and struct bContext are only two examples. Other types fit this pattern as well, such as struct Global.

BKE_main.h

The BKE_* module API functions prototyped in BKE_main.h are shown in Listing 4-12. struct Main is defined in BKE_main.h. The file source/blender/blenkernel/intern/main.c implements the related API.

...

struct Main *BKE_main_new(void);

void BKE_main_free(struct Main *mainvar);

void BKE_main_lock(struct Main *bmain);

void BKE_main_unlock(struct Main *bmain);

void BKE_main_relations_create(struct Main *bmain, const short flag);

void BKE_main_relations_free(struct Main *bmain);

struct GSet *BKE_main_gset_create(struct Main *bmain, struct GSet *gset);

...

Listing 4-12

Example BKE_main.h API prototypes. Function names are in boldface

BKE_context.h

The BKE_* module API functions prototyped in BKE_context.h are shown in Listing 4-13. struct bContext is defined in source/blender/blenkernel/intern/context.c. Related functions are also defined in context.c.

...

/* Context */

bContext *CTX_create(void);

void CTX_free(bContext *C);

bContext *CTX_copy(const bContext *C);

/* Stored Context */

bContextStore *CTX_store_add(ListBase *contexts, const char *name, PointerRNA *ptr);

bContextStore *CTX_store_add_all(ListBase *contexts, bContextStore *context);

void CTX_store_set(bContext *C, bContextStore *store);

bContextStore *CTX_store_copy(bContextStore *store);

void CTX_store_free(bContextStore *store);

void CTX_store_free_list(ListBase *contexts);

/* need to store if python is initialized or not */

bool CTX_py_init_get(bContext *C);

void CTX_py_init_set(bContext *C, bool value);

void *CTX_py_dict_get(const bContext *C);

void CTX_py_dict_set(bContext *C, void *value);

...

Listing 4-13

Example BKE_context.h API prototypes. Function names are in boldface

Summary

In this chapter, we looked at the blenlib and blenkernel modules. We also saw some of Blender’s unit tests, which are helpful in understanding parts of the codebase. Blender uses Google Test for at least some of its testing.

blendlib handles generic utilities in Blender, while blenkernel is more specific to the Blender application. Therefore, blenlib’s functionality, at least in principle, could be used by other similar programs. However, blenlib is tightly coupled to the Blender codebase—more so than, say GHOST, which is not part of the “core” codebase as we saw in Chapter 3.

blenkernel, on the other hand, offers a number of API functions for making needed manipulations on Blender “DNA,” and other state-keeping data structures. We provided a list of the correspondence between makesdna and blenkernel’s files, to illustrate the partial pairing of those modules.

Footnotes

1

An xUnit framework written by Google for C++ (and C) programs.

 

2

Thus, blenlib has utility “mixin” classes, i.e., classes not viable on their own. “Mixins” use multiple inheritance to combine member data and functions between more than one class type.