Photo by Joey Huang on Unsplash
Hi there, here we go again, on the first part (available here, if you missed it) I set up the package structure with assets and migrations, here I will start the real coding with models, controllers, routes and views to handle the real package logic.
How does recording works?
Recordings must be very transparent to the system, the package user should not lose time to integrate recordings into its system. To follow the KISS philosophy, the recording structure should be composed of just a few steps!
Spyhole will record user sessions using RRWeb in the user frontend totally stealth. Recordings will be collected into a collection composed of a minimum events count (this can be set on config) that will be posted to be stored.
The Model
The package models should be in the src/Models folder (the src folder it’s like the Laravel app folder). The model should just “translate” the migration to the Eloquent version. I decided to add some PHPDoc to help the reading and give some IDE support. A little notice: the user recordings come from the frontend and will be stored “as-is”, so I decided to encode the full JSON payload from the frontend to base64 after gzipping it! This operation can be simplified and be transparent to the model usage using eloquent mutators.
<?php
namespace Kalizi\LaravelSpyhole\Models;
use Illuminate\Database\Eloquent\Model;
/**
* Class SessionRecording
*
* @property int id
* @property string path
* @property string session_id
* @property array recordings
* @property string|null user_id
* @package Kalizi\LaravelSpyhole\Models
*/
class SessionRecording extends Model
{
/**
* The table associated with the model.
*
* @var string
*/
protected $table = 'session_recordings';
public function getRecordingsAttribute()
{
return json_decode(gzdecode(base64_decode($this->attributes['recordings'])));
}
public function setRecordingsAttribute($value)
{
$this->attributes['recordings'] = base64_encode(gzencode(json_encode($value)));
}
}“Hello, it’s TDD here!” 🎯
Photo by JESHOOTS.COM on Unsplash
First time I got to test-driven development alone I was like “why the f@*# I have to do this?” and I can’t blame anyone for having the same reaction. But studying, getting in touch with people who regularly use TDD and after tries and tries, you start thinking “well, it wasn’t so bad” (in my case at least).
When I can, and I want, I use TDD in my projects! In the developing world, there are a lot of ways to approach TDD. After lots of tests, I decided to stick with this working way:
- You start projecting the flow of your code without coding: you have to get a clear idea of the feature you want to implement, what it must receive as parameters and it must return as an output.
- You write tests: I learned this technique from Linkedin. At first glance, I thought “What a crap? You didn’t even start coding and you already have to test?” and… well yes, but no. The purpose of writing tests before coding is forcing you deeply thinking how your feature works.
- You run tests: you never wrote code, the tests must fail, so why? Well, it’s to test your engine. Suppose tests pass… there would be a problem, wouldn’t it?
- You write code: that’s the part where you implement your feature. At this point, you should have a clear idea of what you have to write. This doesn’t mean you can’t change your ideas about some features but in general, thinking this way should help you.
- You run tests: and this step will tell you if you did great or must fix something.
Another pro is that it helps with splitting tasks: you can write code, while another person write tests.
Analyzing Spyhole, the package should have a route where to POST data, a Controller to handle the storing and a Request to validate data.
The Controller should store the recordings from the controller with its path. Every time the frontend stores newer recordings, the controller will enqueue the newer recordings (this means that once a recording session is created, its identifier should be returned to the frontend). The package can be configured to store user ID with recordings, to be tested too. Last but not least, session ID from the Session class for tracking, another test to do.
From this analysis, I thought the tests to do where 4:
- A test to check that the first request creates a new recording.
- A test to check that a new request enqueues recordings to the existing one.
- A test to check if the user ID gets stored if the option is enabled.
- A test to check if the session ID gets stored if the option is enabled.
Before starting, Laravel testing doesn’t start sessions with PHPUnit as default behaviour, should be enabled via setup.
protected function setUp(): void
{
parent::setUp();
$kernel = app('Illuminate\Contracts\Http\Kernel');
$kernel->pushMiddleware('Illuminate\Session\Middleware\StartSession');
}Test #1: create a new recording
The purpose of this test is to post some fake data and check that:
- the session ID is not stored in favour of a fake ID.
- the fake ID is kept between multiple calls.
- the user ID is not tracked.
- the recording is stored and its ID is returned.
<?php
/**
* This test check if can correctly store a request with recording.
* @test
*/
public function can_store_first_recording_request()
{
$this->assertFalse(config('laravel-spyhole.track_request_session_id'));
$requestData = [
'frames' => [
// some example data
[
'timestamp' => now()->unix(),
'data' => [
'x' => 0,
'y' => 0,
'type' => 0
]
]
],
'path' => '/',
];
$response = $this->json(
'POST',
route('spyhole.store-entry'),
$requestData
);
$response->assertSuccessful();
$response->assertJsonStructure([
'success',
'recording',
]);
$this->assertTrue($response->json('success'));
$this->assertIsNumeric($recordingId = decrypt($response->json('recording')));
$this->assertDatabaseHas(
'session_recordings',
[
'id' => (int)$recordingId,
'path' => '/',
'recordings' => base64_encode(gzencode(json_encode($requestData['frames']))),
'user_id' => null,
]
);
$recording = SessionRecording::find((int) $recordingId);
$this->assertNotEquals($this->app['session']->getId(), $recording->session_id);
}About this test:
- I expect that the request should contain a key
frameswith recordings data from RRWeb and a keypathabout the current user path. - I expect that the response returns a key
successto check that everything went well and a keyrecordingwith the encrypted ID. - I expect that frames are encoded and gzipped from the initial JSON.
- I expect that the session ID stored is randomly generated and different from the real ID.
Test #2: enqueue recordings to an existing record
The purpose of this test is to post some fake data and an encrypted recording ID and check that
- the user ID is still not tracked.
- the recordings are merged.
<?php
/**
* This test check if can correctly store frames into the same row of an existing session.
* @test
*/
public function can_store_frames_for_a_started_session()
{
$recording = new SessionRecording();
$recording->recordings = [
[
'timestamp' => now()->unix(),
'data' => [
'x' => 0,
'y' => 0,
'type' => 0
]
]
];
$recording->path = '/';
$recording->session_id = Str::uuid();
$recording->save();
$requestData = [
'frames' => [
[
'timestamp' => now()->unix(),
'data' => [
'x' => 0,
'y' => 0,
'type' => 0
]
]
],
'path' => '/',
'recording' => encrypt($recording->id),
];
$response = $this->json(
'POST',
route('spyhole.store-entry'),
$requestData
);
$response->assertSuccessful();
$response->assertJsonStructure([
'success',
'recording',
]);
$this->assertTrue($response->json('success'));
$this->assertIsNumeric($recordingId = decrypt($response->json('recording')));
$this->assertDatabaseHas(
'session_recordings',
[
'id' => (int)$recordingId,
'path' => '/',
'recordings' => base64_encode(gzencode(json_encode(array_merge(
$recording->recordings,
$requestData['frames']
)))),
'user_id' => null,
]
);
}About this test:
- I expect that the request should contain a key
frameswith recordings data from RRWeb, a keypathabout the current user path and a keyrecordingwith the encrypted ID from the previous request. - I expect that the response returns a key
successto check that everything went well and a keyrecordingwith the encrypted ID. - I expect that recordings from this request are encoded and gzipped, then merged with the previous.
Test #3: store recordings with currently logged in user
The purpose of this test is to post some fake data and check that
- the user ID is tracked.
<?php
/**
* This test check if can correctly store the user id while the configuration option is enabled.
* @test
*/
public function can_store_recording_with_logged_in_user()
{
config()->set('laravel-spyhole.record_user_id', true);
// Mock a fake user
$user = new FakeUser();
$user->id = rand(0, 1000);
Auth::shouldReceive('user')->andReturn($user)->once();
$requestData = [
'frames' => [
// some example data
[
'timestamp' => now()->unix(),
'data' => [
'x' => 0,
'y' => 0,
'type' => 0
]
]
],
'path' => '/',
];
$response = $this->json(
'POST',
route('spyhole.store-entry'),
$requestData
);
$response->assertSuccessful();
$response->assertJsonStructure([
'success',
'recording',
]);
$this->assertTrue($response->json('success'));
$this->assertIsNumeric($recordingId = decrypt($response->json('recording')));
$this->assertDatabaseHas(
'session_recordings',
[
'id' => (int)$recordingId,
'path' => '/',
'recordings' => base64_encode(gzencode(json_encode($requestData['frames']))),
'user_id' => $user->id,
]
);
}
class FakeUser implements Authenticatable
{
/**
* @var int $id Fake Identifier
*/
public $id;
public function getAuthIdentifierName(): string
{
return 'test';
}
public function getAuthIdentifier(): int
{
return $this->id;
}
public function getAuthPassword(): string
{
return 'password';
}
public function getRememberToken(): string
{
return '';
}
public function setRememberToken($value)
{
}
public function getRememberTokenName(): string
{
return '';
}
}About this test:
- I expect that the request is exactly like the previously tested.
- I expect that the config has the user tracking turned on.
- I expect the user ID to be stored in the database record.
- The authentication is mocked via the
Auth::shouldReceive, this will force the usage of theAuth::usermethod.
Test #4: store the session ID from Laravel Session
The purpose of this test is to post some fake data and check that
- the real session ID is tracked.
<?php
/**
* This test check if can correctly store the user id while the configuration option is enabled.
* @test
*/
public function can_store_correct_session_id()
{
config()->set('laravel-spyhole.track_request_session_id', true);
$requestData = [
'frames' => [
// some example data
[
'timestamp' => now()->unix(),
'data' => [
'x' => 0,
'y' => 0,
'type' => 0
]
]
],
'path' => '/',
];
$response = $this->json(
'POST',
route('spyhole.store-entry'),
$requestData
);
$response->assertSuccessful();
$response->assertJsonStructure([
'success',
'recording',
]);
$this->assertTrue($response->json('success'));
$this->assertIsNumeric($recordingId = decrypt($response->json('recording')));
$this->assertDatabaseHas(
'session_recordings',
[
'id' => (int)$recordingId,
'path' => '/',
'recordings' => base64_encode(gzencode(json_encode($requestData['frames']))),
'user_id' => null,
'session_id' => $this->app['session']->getId()
]
);
}About this test:
- I expect that the request is exactly like the previously tested.
- I expect that the config has the session tracking turned on.
- I expect the session ID to be stored in the database record.
Test #5: the fake session ID is kept between multiple calls
The purpose of this test is to post some fake data twice to different paths and check that
- the fake session ID is tracked between them.
<?php
/**
* This test check if the fake session id is kept between calls.
* @test
*/
public function can_store_recordings_keeping_generated_session_id()
{
$this->assertFalse(config('laravel-spyhole.track_request_session_id'));
$requestData = [
'frames' => [
// some example data
[
'timestamp' => now()->unix(),
'data' => [
'x' => 0,
'y' => 0,
'type' => 0
]
]
],
'path' => '/',
];
$response = $this->json(
'POST',
route('spyhole.store-entry'),
$requestData
);
$response->assertSuccessful();
$response->assertJsonStructure([
'success',
'recording',
]);
$recordingId = decrypt($response->json('recording'));
$recording = SessionRecording::find((int) $recordingId);
$this->assertNotEquals($this->app['session']->getId(), $recording->session_id);
$requestData['path'] = '/path_changed';
$secondResponse = $this->json(
'POST',
route('spyhole.store-entry'),
$requestData
);
$secondResponse->assertSuccessful();
$response->assertJsonStructure([
'success',
'recording',
]);
$this->assertDatabaseHas(
'session_recordings',
[
'id' => (int)$recordingId,
'path' => '/',
'recordings' => base64_encode(gzencode(json_encode($requestData['frames']))),
'user_id' => null,
'session_id' => $recording->session_id
]
);
}About this test:
- I expect that the first request is accepted.
- I expect that the second request is accepted and holds the same session ID.
The controller logic 📔
Now it’s kinda fun because the controller logic is inferred from the tests code.
Data payloads have the same structure that can be validated through a Request. The Request authorization can be provided just if the content of the request is passed as JSON and this can be achieved using the wantsJson in the authorize.
public function authorize(): bool
{
return $this->wantsJson();
}For the body, we want to validate the three fields identified.
public function rules(): array
{
return [
// recording frames
'frames' => 'required|array',
// previous recording id (encrypted)
'recording' => 'sometimes|string',
// recorded path
'path' => 'required|string',
];
}Now that the request is ready, just focus on the controller.
<?php
namespace Kalizi\LaravelSpyhole\Http\Controllers;
use Illuminate\Http\JsonResponse;
use Illuminate\Routing\Controller;
use Illuminate\Support\Facades\Auth;
use Illuminate\Support\Str;
use Kalizi\LaravelSpyhole\Http\Requests\StoreEntryRequest;
use Kalizi\LaravelSpyhole\Models\SessionRecording;
use Symfony\Component\HttpKernel\Exception\NotAcceptableHttpException;
class EntryController extends Controller
{
public function store(StoreEntryRequest $request): JsonResponse
{
$recordingId = null;
if ($request->has('recording')) {
$recordingId = (int)decrypt($request->get('recording'));
if (SessionRecording::whereId($recordingId)->count() === 0) {
throw new NotAcceptableHttpException();
}
}
if (config('laravel-spyhole.track_request_session_id')) {
$sessionId = $request->session()->getId();
} else {
if (session()->has('spyhole_session_id')) {
$sessionId = session()->get('spyhole_session_id');
} else {
do {
$sessionId = Str::uuid()->toString();
} while (
SessionRecording::whereSessionId($sessionId)->count() > 0 &&
$sessionId !== $request->session()->getId()
);
session()->put('spyhole_session_id', $sessionId);
}
}
$userId = null;
if (config('laravel-spyhole.record_user_id')) {
$user = Auth::user();
$userId = $user ? $user->getAuthIdentifier() : null;
}
if ($recordingId === null) {
$recording = new SessionRecording();
$recording->session_id = $sessionId;
$recording->user_id = $userId;
$recording->path = $request->get('path');
$recording->recordings = $request->get('frames');
} else {
$recording = SessionRecording::wherePath($request->get('path'))
->whereId($recordingId)
->first();
if ($recording === null) {
throw new NotAcceptableHttpException();
}
// Merge frames from the same session
$recording->recordings = array_merge(
$recording->recordings,
$request->get('frames')
);
}
$recording->save();
return response()->json([
'success' => true,
'recording' => encrypt($recording->id),
]);
}
}The controller logic is very clean and you shouldn’t have problems reading it. Every time a data is invalid, a Not Acceptable Exception is thrown.
Now that the controller is implemented, you can run the test suite!
$ ./vendor/bin/phpunit --testdox
PHPUnit 8.5.14 by Sebastian Bergmann and contributors.
Runtime: PHP 8.0.1
Configuration: ./phpunit.xml.dist
Warning - The configuration file did not pass validation!
The following problems have been detected:
Line 25:
- Element 'log', attribute 'charset': The attribute 'charset' is not allowed.
- Element 'log', attribute 'yui': The attribute 'yui' is not allowed.
- Element 'log', attribute 'highlight': The attribute 'highlight' is not allowed.
Test results may not be as expected.
Error: This version of PHPUnit does not support code coverage on PHP 8
Store (Kalizi\LaravelSpyhole\Tests\Http\Store)
✔ Can store first recording request 450 ms
✔ Can store recordings keeping generated session id 39 ms
✔ Can store frames for a started session 38 ms
✔ Can store recording with logged in user 45 ms
✔ Can store correct session id 32 ms
Time: 16.26 seconds, Memory: 24.00 MB
OK (4 tests, 27 assertions)Tests passed! 🎯
The route 🗾
Controller and Request are ready and tested, but right now, nobody knows about them because they aren’t exposed. Bounding routes and controller it’s really easy and works as you would do in a Laravel Project.
It all starts with the routes file in src/routes.php where the route can be declared using the Facade Route:
use Kalizi\LaravelSpyhole\Http\Controllers\EntryController;
Route::post('/spyhole-api/record', [EntryController::class, 'store'])->name('spyhole.store-entry');I decided to pick named routes so that every route can be accessed via the route helper and URLs can be changed if needed.
But the package doesn’t know that we have routes, the routes file must be advertised on the Service Provider, this way:
$this->loadRoutesFrom(__DIR__ . '/routes.php');And that’s it, routes are ready to go!
Next step?
Well, now that backend’s ready, what’s missing is the frontend, on the next step I’ll be building the JS to embed on any page to record everything and connecting it to the just created controller!
Stay tuned for the complete series and if you want, take a moment to leave a comment about how you would do the projecting or if you would have changed something in this step! ☕️
✔️ Improving request validation ✔️
To build the package faster, while reading the first version of the Request class, I just put the “array” validation rule for the recording's key. It just works. But… it’s good? No. This is the opposite of security by design.
Let’s improve this.
RRWeb uses Mutation Watcher to serializes the DOM and its changes, so part of the request payload can be “reversed” from the serialization documentation.
So I decided to pick a little set of recordings (400 events) to extract a schema from them using JSONSchema.net.
JSON Schema Extraction
JSON Schema is a vocabulary that allows you to annotate and validate JSON documents.
Laravel hasn’t built-in JSON Schema validation, you can surely use an additional package, but Laravel Validation already provides everything I need. So why use JSON Schema? To extract a pattern.
JSON Schema as a Graph
The root in the graph it’s described with the type “array”, ok so far. Now expand it, it basically has only the items we want to validate, additionalItems is added to explicit that everything different from the first key is valid. Now focus on each item. This schema is really simple, it just says that each item, by inferring the pattern from all given data is composed of:
type: an integer. Its rule should be'frames.*.type' => 'required|integer'.timestamp: a UNIX time string. Laravel doesn’t provide timestamp validation out of the box. By the way, a rule can be added via a function. Its rule should be:
'frames.*.timestamp' => [
'required',
'integer',
function ($attribute, $value, $parameters) {
try {
return Carbon::createFromTimestamp($value)->isCurrentHour();
} catch (\Exception $invalidTimestampException) {
return false;
}
}
],additionalProperties: asadditionalItemsthis would accept any properties not included in the schema, but can just be skipped.data: here’s the big problem. This array key is extremely mutable since RRWeb has lots of data to post: data about the browser width and height, serialized data about nodes and their changes, the user events, and all this data are different, so the only rule that can be used is “array”:'frames.*.data' => 'required|array'.
Remember that data aren’t stored in the database as-is, they’re gzipped and base64 encoded. This edit can be considered a minor fix and shouldn’t break tests, so running them shouldn’t give any problem!