-
Notifications
You must be signed in to change notification settings - Fork 4
/
README
1842 lines (1480 loc) · 89.4 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
fenris - program execution path analysis tool
Copyright (C) 2001, 2002 by Bindview Corporation
Portions copyright (C) 2001, 2002 by their respective contributors
Developed and maintained by Michal Zalewski <lcamtuf@bos.bindview.com>
==========================================================================
Sections in this document:
[0x00] What is Fenris? [0x01] How does it work?
[0x02] How can I use it? [0x03] Project limitations
[0x04] Security issues / forensics [0x05] Tracing mechanism
[0x06] Bug reporting and other feedback [0x07] Known bugs / TODO
[0x08] Thanks and credits [0x09] Further reading
URL: http://razor.bindview.com/tools/fenris/
Devel: http://lcamtuf.coredump.cx/fenris/devel.shtml
Demo: http://lcamtuf.coredump.cx/fenris/demos.shtml
Mailing list: echo subscribe fenris-devel | mail listar@7thguard.net
http://lists.7thguard.net/fenris-devel/
Portions of this program use code derived from libi386 library, a part of
project 'bastard', and from GNU binutils. See libdisasm/00-READ_FIRST for more
details. Portions of 'dress' utility inspired by klog's code from Phrack 56.
For alternative programs of this kind and other promising projects, be sure to
visit http://lcamtuf.coredump.cx/fenris/other.txt .
*************************************************************************
NOTE: It starts to be more and more difficult to describe this project in few
simple words. It started as something you'd call "not your typical tracer",
and quickly turned into a suite of several different tools for binary
analysis, visualization, debugging, forensics, and so on. You can use some of
its features and never know about others, but it is my duty to document all of
them right here - so be patient and skip parts that do not seem very
interesting to you :-) This document is long due for a general rewrite.
*************************************************************************
[0x00] What is Fenris?
----------------------
First, I'd like to explain why I even decided to develop Fenris, and what
makes it different. It is a lengthy and often pathetic section, but probably
a good summary for those of you who wonder...
Code analysis is not limited to debugging, quality assurance or security
audits. Understanding and handling file formats or communication protocols
used by proprietary solutions, a problem that many corporations face when they
decide to change their base software platform from one, obsolete or
insufficient solution to another, perhaps more suitable, is a task that can
consume long months and millions of dollars, especially when any misjudgment
or misinterpretation is made. Because of that, accurate and complete
information about existing solutions has to be obtained and evaluated in
a timely manner. This project is an attempt to fill the gap between currently
used tools by providing a freely available program analysis utility, suitable
for black-box code audits, algorithm analysis, rapid reconnaissance in
open-source projects, tracking down bugs, evaluating security subsystems,
performing computer forensics, etc.
This program does not automate the process of auditing, and does not favor any
particular use. Instead of that, it is intended to be a flexible and universal
application that will be a valuable solution for many advanced users. While
functional, it is probably not tested sufficiently, there are many issues to
fix, several known bugs, some portability problems. It is being released
primarily to get user feedback, comments, and, most important, to request
development support, as my resources are very limited, both in terms of
available time and development platforms. This project is and will be
distributed as a free software, regardless of projected use, accompanied by
complete sources, under the terms and conditions of GPL. Why do you might need
this code? Well, there are few reasons...
Human beings are, so far, the best code analysts. Unlike computer programs,
they have imagination, ability to build synthetic abstract models, and yet to
observe and analyze smallest details at the same time. Functionality is often
being described as "doing what the program is supposed to do", security as
"doing what the program is supposed to do and nothing more". While it might
sound funny, that is the most general and complete definition we have. In most
real-life scenarios only humans really know what are their expectations.
Building strict formal models of our expectations does not necessarily mean
that models themselves are flawless, and is very time-consuming. Then, even
with such models, validating the code is not always possible, due to its
computational complexity. That is why real, live programs (not including some
critical developments) do not have such models, do not follow any particular
coding guidelines, and cannot be formally examined without human judgment.
Unfortunately, humans are also highly inaccurate and very expensive. They work
slowly, and better results can be achieved by hiring better specialists and
performing more careful audit. And after all, even the best expert can
overlook something in complex, hard to read code. It is almost impossible for
human to perform an accurate audit of a large, complex, heterogeneous project
written e.g. in C - like Sendmail, BIND, Apache - and provide results in
reasonable time. Things get even worse when humans try to understand
algorithms and protocols used by complex closed-source black box solutions.
They are simply too slow, and not always able to make accurate guesses about
dozens of complicated, conditional parameter passes and function calls before
final action is taken. While it might sound surprising, human-driven code
audit is very similar to playing chess - it is a general analysis of possible
states, way too many to be implicitly projected by our conscience, a result of
experience, knowledge, some unparalleled capabilities of human brain, and
luck. It is also a subject to false moves and misjudgment. And there are maybe
just a few hundred excellent players.
As for today, freely and commercially available audit tools both use two
opposite approaches. First approach tends to minimize human role by automating
the review of source code. Source code analysis methods are good in spotting
known, repeatable static errors in the code - such as format string
vulnerabilities. On the other hand, static tools are not able to trace and
analyze all possible execution paths of complex application by simply looking
at its source. The reason for inability to follow all execution paths lies
deeply in the foundations of modern computation theory, and one of its aspects
is known as "the halting problem". Speaking in more general terms, in many
cases (such as complex software, or even underlying operating system), the
amount of medium needed to store all possible states of a complex program
exceeds significantly the number of particles in the universe; and the amount
of time needed to generate and process them sequentially is greater than the
lifetime of our universe, even having a machine that works with the speed of
light. This might be changed by the development of new computation models,
such as quantum computing, or by creating mathematical models that allow us to
make such problems non-polynomial - but for now, we are far from this point,
and static analysis is restrained in many very serious ways, even though many
software suppliers tend to market their products as the ultimate, 100%
solutions. Subtle, complex, conditional dynamic errors, such as privilege
dropping problems, input-dependent table overflows in C and many other issues
usually cannot be detected without generating a completely unacceptable number
of false positives. This kind of software is highly dependent on coding style,
and specific notation or development practices might render them less
efficient - for example, automated audit utilities can usually detect problems
like insecure call to strcpy() function, but will very likely not notice
insecure manual copy in do-while loop. The truth is, for programs that do not
have previously built formal models, static auditing utilities look for known,
common problems in known, common types of code in a very limited scope.
Another issue is the applicability of this approach to algorithm analysis
tasks. In the domain of automated audit tools, this problem is "reduced" to
building a formal model of program behavior, or, more appropriately,
generating certain predictive statements about the code. While there are very
interesting developments in this direction, such as the work of professor
Patrick Cousot, it is very difficult to make any detailed, accurate and
abstract enough run-time predictions for complex source code that has any
immediate value in the analysis of unknown algorithm.
Last but not least, static analysis of sources can be deployed only when the
source code is available, which does not have to be the case. This approach is
a subject to many shortcomings, tricky assertions, and is a technique of
strictly limited capabilities. This is, of course, not to dismiss this method
- but to demonstrate that this much favored approach is not flawless and how
much it needs to be accompanied with auxiliary methods.
The second approach to be discussed here is based on a dynamic run-time
program analysis. This method is usually used to provide the user with
information about actual program execution path, letting him make decisions on
which path to follow and giving him free will to draw any conclusions and
perform all the synthetic reasoning. This method is applied to a live binary
executed in real-time and is based on monitoring syscalls (strace), libcalls
(ltrace) or functions (xtrace); in certain cases, breakpoint debuggers, such
as gdb, can be used, however it is usually not feasible to use them to perform
anything more than in-depth analysis of a very small portion of program
functionality. Usually, such analysis provides a very useful information on
what is happening, and this information is provided in uniform, reduced-output
form. A careful auditor can analyze program behavior and find interesting or
potentially dangerous run-time conditions. By monitoring how a given
application interacts with external world, he (or she) can determine whether
some other conditions can be triggered and eventually explore them by
examining sources or re-running the program. Advantages are enormous, as such
software enables the auditor to spot very subtle errors in code that "looked
good", to observe actual execution, not to try to figure it out, and to find
or trace down not obvious or non-schematic vulnerabilities. Run-time trace
tools are primarily used for fast reconnaissance tasks and for tracing down
notorious errors that are not clearly visible in the source, significantly
reducing the time of such operations. There are, however, serious drawbacks
related to this method. First of all, known tracing tools do not provide the
complete information. They will detect strcpy() call, but won't report if
exactly the same functionality has been implemented from scratch by the author
of given program. And, in some cases, the amount of produced data can be
enormous, and because of its completely unstructured character, it makes the
observation of overall execution vector almost impossible. Two most important
problems are: correlating trace data with actual code, and determining what
occurred in the "dark matter" between two lines of trace output.
There are some attempts to combine both approaches - run-time evaluation and
source code analysis - such as Purify or many other commercial development
support products. Unfortunately, they all feature a limited set of
capabilities that need development-side or compilation-time support and are
not really suitable for comprehending black box solutions or performing
a general analysis. Most of them are targeted for dynamic memory debugging and
code / memory profiling.
While not mentioned above, there is also another approach to black-box code
- high-level decompiler. However, the complexity of modern compilers makes it
very difficult to develop an effective C decompiler or similar utility, and
there are only a few (two?) projects available to accomplish it, all of them
not able to deal with too complex or optimized code. Finally, there is no
guarantee that generated output code will be any help in comprehending the
program. For now, this approach remains almost purely theoretical, and I am
not aware of any auditors using it extensively. Why? Well, here's an example
of decompiled, mildly optimized code *with* some symbolic information:
http://www.backerstreet.com/rec/ex386/hdgO.rec . One may argue it is less
readable than cross-referenced disassembly.
This project, Fenris, is named after the monstrous wolf, son of the Norse god
Loki. It is not the ultimate answer to all questions, not a solution for all
problems, and under no circumstances is intended to replace other tools and
techniques. On the other hand, it makes one step forward compared to other
tools, trying to support the auditor and to make his work much more effective.
This is accomplished by combining a number of techniques, including partial
run-time decompiler, stateful analysis, code fingerprinting, I/O analysis,
high-level visualization layer, traditional interactive debugger features and
run-time code modification capabilities. The goal is to provide a very
detailed trace information, and, at the same time, to provide data suitable to
build a model of program behavior more quickly and in more convenient way.
Fenris is not supposed to find vulnerabilities or bugs, or to guess algorithms
or describe protocols. It is supposed to report and analyze the execution path
- detect and describe functional blocks, monitor data flow in the program,
marking its lifetime, source, migration and destination, analyze how functions
work and what conditions are evaluated. At the end, it can deliver you an
execution model of traced program (or arbitrarily chosen portion of it, if
complete trace results in too much noise or irrelevant information), and hint
you how this model can change in different conditions. Fenris does not need
source codes of analyzed application, but obviously does not keep the auditor
from using them.
For many users, Fenris might be a new tool or tools, for others - just
a command-line replacement or addition to strace, ltrace, gdb or similar
applications (there's a brief list of other nice tools in doc/other.txt). And
that's the idea - to build a tool that is simple, reusable, but also precise
and smart. It is supposed to have advantages over other tools, but not to be
an ultimate replacement or the final solution. Some users can just use very
specific features, such as automated function fingerprinting, and use
companion tools instead of the main program.
The core component of Fenris works on real-time tracing level, analyzing
program structure and interaction with libc and OS, detecting and describing
parameters, tracing buffer sizes and such. First layer processing produces
text-format output stream, which can look like that:
8617:00 setuid (0)
8617:01 [L] SYS setuid (0) = 0
8617:00 ...return from libc = 0
8617:00 <0x80123456> cndt: conditional block +10 skipped
8617:00 local fnct_34 (bfffb01a "$something?" )
8617:00 + bfffb01a is bfffb01a:1000 <off 0> (created by create_buffer)
8617:00 last input: my_function:read from /etc/config.cfg
8617:01 __libc_open64 (l/bffff80d "onefile", 3)
8617:02 [L] SYS open (bffff80d "onefile", O_RDWR|O_CREAT) = 4
8617:02 + 0xbffff80d first seen in fnct_4:open, size 20
8617:02 last input: handle_net:recv <from 195.117.3.59:25>
8617:01 ...return from libc = 4
8617:00 ...return from fnct = <void>
8617:00 Function has written non-local memory:
8617:00 * local object irc_servers:123 (0x80654321)
8617:00 * unknown address 0x40123456
This core code can be also used in conjunction with Aegir, a gdb-alike
assembly language level interactive debugger with some interesting
capabilities, such as very precise software break- and watchpoints, full
integration with Fenris and its high-level structures, code fingerprinting and
many other features. Or, if you prefer, nc-aegir, a SoftICE-alike text-mode
debugging GUI. So, Fenris can be used either as an advanced version of *trace
utilities, or as a command-line step-by-step debugger. Because Fenris does not
rely on BFD for any critical tasks, you should be able to work with tweaked
binaries, such as a binary coded with burneye, with no problems:
$ gdb ./startwu
"./startwu": not in executable format: File format not recognized
$ objdump -d ./startwu
objdump: ./startwu: File format not recognized
$ ./fenris -W /tmp/aegir-sock -X 5 ./startwu &
$ aegir /tmp/aegir-sock
...
[aegir] disas
05371035: pushl 0x5371008
0537103b: pushf
0537103c: pusha
0537103d: movl 0x5371000,%ecx
05371043: jmp $0x5371082
05371048: popl %esi
05371049: movl %esi,%edi
For more information, plus an example of how to remove certain
debugging traps please refer to doc/be.txt.
For Aegir, there is some bare-bone API specification in doc/debug-api.txt, and
a description of all available commands in this README. The code is mostly
functional, but, as this is a newly introduced component, it is reasonable to
expect some bugs. Aegir session can look like this:
Cur. time : Wed May 22 00:31:09 2002
Executable: ./a.out
Arguments : <NULL>
[aegir] step
>> Singlestep stop at 0x80483b0 [_start].
080483b0 [_start]: xorl %ebp,%ebp
[aegir] next
At 0x80483b0, continuing to next output line...
20394:00 L memset (8049660, 0, 100) = 8049660
20394:00 + g/8049660 = local buf
20394:00 + g/8049660 = local buf
20394:00 \ new buffer candidate: 8049660:100 (buf)
20394:00 \ buffer 8049660 modified.
20394:00 >> New line stop at 0x400a5b0c [memset+68].
080484bb [main+23]: addl $0x10,%esp
[aegir] info buf
Name 'buf' has address 0x08049660.
+ 8049660 = 8049660:100 <off 0> (first seen in L main:memset)
last input: L main:memset
[aegir] fdinfo 0
+ fd 0: "/dev/tty6", origin unknown
[aegir] wwatch 0x08049660 0x08049670
Breakpoint #0 added.
[aegir] list
00: stop on write 0x8049660-0x8049670.
[aegir] fprint 0x080485a4
Matches for signature CC6E587C: printf, wprintf
[aegir] call
At 0x80484bf, continuing to next local call...
08048492 [funkcjadwa+6]: call $0x804849c <funkcjasiedem>
21364:02 local funkcjasiedem ()
21364:02 + funkcjasiedem = 0x804849c
>> Local call to 0x804849c reached at 0x8048492 [funkcjadwa+6].
[aegir] back
Local function calls history (oldest to most recent calls):
From 80484bf [main+11]: fnct_1 [funkcjadwa] 804848c, stack bffffa64 -> ...
From 8048492 [funkcjadwa+6]: fnct_2 [funkcjasiedem] 804849c, stack bfff...
The GUI version of Aegir, nc-aegir, works basically the same way, but provides
an organized debugging screen with register, memory and code views, integrated
Fenris output view, and automatic control over Fenris parameters. For
a snapshot of nc-aegir layout, please go to
http://lcamtuf.coredump.cx/fenris/nc-aegir.gif :-)
Core output is a descriptive trace of program activity. For a larger project,
this trace might have many megabytes, and, to avoid problems reading it, it
should be either delimited to a relevant structural portion of code that is
being analyzed right now, or should be transformed in a different model of
execution path. One of such trivial transform tools is called 'ragnarok'. It
is able to convert first layer output into a graphical html model that
summarizes program structure, marks data migration, I/O points, conditional
points, and, on request, provides detailed information about single functions.
Ragnarok is hardly an integral component of Fenris - it is merely a parser
that transforms text output into html. Any other presentation schemes or
destination formats can be easily achieved because of easily parsable output
generated by Fenris. We're looking towards making ragnarok more user friendly,
by using OpenDX or another interactive data browsing mechanism instead of
static contents, but as of today, Ragnarok output consists of five sections,
or views:
a) "program flow": a table, or, in the future, a browsable graph
with horizontal lines (rows) representing functions and vertical lines
(columns) representing buffers and I/O sources (Y axis is time). When a
function uses one of the buffers or interacts with I/O source,
intersection of this object and function is marked in a way dependent
on the nature of this operation. Conditional expressions evaluated
during program execution are also marked. Data migration is marked.
Additional table section outlines interaction with data sources
(file descriptors).
b) "function view": a description of a single function call: current
parameters, modified memory, return code, inside-function conditions,
function trace, etc - everything intended and providing some
useful shortcuts to follow nested calls.
c) "function summary": a summary of all calls of this particular function,
useful for determining the purpose and range of accepted parameters
of given function.
d) "buffer view": a history of all modifications and I/O ops applied
to a single buffer.
e) "I/O view": a history of all I/O operations performed on a single
file descriptor.
The main idea of Ragnarok, as you probably can imagine, is to provide five
different views of program activity, each of them can be used separately for
different purposes. For example, buffer view can be very useful for
vulnerability research, and function summary view will be an invaluable source
of reverse engineering information. Ragnarok basically consists of five
different and loosely connected parts.
Ragnarok is text-browser friendly, but until OpenDX or a similar solution is
deployed, be warned that reading outputs for complex programs might be rather
unpleasant. As HTML is not necessarily the most useful way of presenting huge
amounts of formatted data, be prepared to get few megs of html for less than
one meg of trace output (still, -R option is a neat workaround; thanks to
negative, we also provide 'split' utility to convert Ragnarok output file into
several separate html pieces).
Fenris components provide some other interesting features, as well. For
example, it is able to manage a database of MD5 fingerprints of functions.
With a companion utility provided with Fenris, you can even recover symbol
tables for stripped static binaries and then use your favorite debugger:
Before:
0x804811c: call 0x8054ad4
After:
0x804811c: call 0x8054ad4 <libc_start_main / xdrrec_create>
By default, only libc functions are placed in the database, but it is fairly
easy to archive user's own functions there. For the core tracer, if any
unknown function is found (e.g. in static binary with no symbols), quick
matching is performed and all possible guesses are reported back. This makes
work much easier, knowing that fnct_123 is actually fprintf, and fnct_124
- vfprintf:
25281:02 local fnct_12 (0, 0, 0)
25281:02 + fnct_12 = 0x804c868
25281:02 # Matches for signature 42BA3CA4: libc_init_secure
I must say I'm pretty disappointed to find out that IDA disassembler for
Windows features somewhat similar mechanism for function fingerprinting =) On
the other hand, IDA is something completely different, plus I believe my
implementation is simply more straight and accurate :) And they don't have
symtab recovery!;P But enough ranting.
Other interesting features include text buffer auto-detection, buffer size and
modification tracking and such. Core tracer provides some limited code
modification capabilities, so that you can, for example, NOP-out debugger
detection code, checksum verification, or some fork() you don't want to
happen. You can also achieve much more advanced control with the companion
debugger.
(On a side note:) This is very advisable to eliminate fork()s and trace one
execution branch at once, as Linux does not provide a reliable mechanism for
attaching to the forked child immediately. There are some tricks to do it
(such as manually inserted jmp-to-self loop that is restored later when the
process is attached), but it is not implemented in Fenris at this moment. I'm
one of guys who believe that ptrace() should be taken out and shot, and who
loves projects like the-dude [http://the-dude.sourceforge.net], but as for
today, Fenris sticks with the default Linux tracing facility, which, by the
way, affects its performance.
Ok, that was a quick introduction. The reasons why you might want this project
are exactly the same as the reasons you might find it useless :-) It is very
verbose, built around the idea of automated low-level analysis, and not all
people will find it useful. But even if you hate it so far, I encourage you to
research alternative you might have never heard of and support other promising
developments - please read doc/other.txt for more information.
So let's dive into more technical aspects. This README is not intended to
describe all features of this project or to provide complete documentation, at
least not for now, but it should provide a reasonable level of detail.
I believe the project is rather intuitive once you get the idea :-)
[0x01] How does it work?
------------------------
Fenris is far from being complex. The key to its simplicity is the fact it
traces one execution path instead of trying to predict all possibilities. It
is all about detecting, reporting and analyzing low-level assembly language
constructions that are known to represent specific operations. Current version
of Fenris is developed to work fine with i386 executables on Linux, and to
understand code generated by GNU C compiler, or other compilers that use GCC
framework - GNU C++, GNU Ada Translator, GNU Prolog, possibly GNU Fortran. On
the other hand, there is no reason why it cannot be ported to support other
high-level languages, compiler frameworks or architectures - and one of main
reasons for releasing the code is having this process of porting started.
The most significant drawback of using universal real-time tracers such as
ltrace or strace is a lack of abilities to observe internal program structure,
function calls, conditional execution, etc - "black matter" mentioned earlier.
Unfortunately, for many programs, this black matter, not I/O manifestations,
is critical for understanding algorithms. I/O activity is just a shadow of
something that cannot be seen. There are certain programs, such as xtrace,
that are capable of providing this additional information, but their
applications are very limited - they are not suitable for black-box code, and
provide a very basic, unstructured information. Another issue is that even
such a basic analysis as what strace gives you - just describing known
parameters of known syscalls - is not really integrated with any interactive
tools, so when you work with gdb, you're forced to figure out every single
mindless detail that could easily be parsed for you, and it takes much longer
to get the idea of what is going on.
Fenris tries to change this situation in many ways. Others than mentioned
earlier (ragnarok, partial tracing):
a) recognizing internal function calls and assigning unique names to them
even if original name cannot be determined,
b) automatically counting the number of passed parameters and recognizing
them appropriately, so the purpose of given function and its domain can
be determined easily; used mechanism can handle such a complex
expressions as "myfunction(otherfunction(7*sin(1))*2,17*i,val?10:0)" and
handle parameters properly, even in the code generated with high
optimization settings; parameter detection includes automatic
search for text entities and displaying them properly,
c) providing appropriate information about function nesting (both using
visual indentation and numbers), so the structure of program is clearly
visible and certain layer of processing can be extracted and separated
easily,
d) detecting shared code by comparing MD5 fingerprints for functions in
statically linked binaries; this way, in many black-box solutions,
unique functions can be distinguished from common library functions
in a quick way; it also enables the auditor to maintain a database
of already identified functions and have their names displayed every
time given code is called,
e) stateful tracing; Fenris maintains an internal map of used memory,
opened files, etc. Whenever a file descriptor or a buffer is
involved in any operation, its current state, information about its
origins, etc, is provided.
f) buffer detection and tracing; buffer sizes and contents is being
traced; both modifications performed by known library functions and
syscalls, and manual access performed by unknown functions is
reported (assembly-language level analysis is used); additional memory
analysis is deployed to keep tracks of buffer lifetime and location
(this includes local buffers passed to nested functions, too),
g) certain known library and system calls are handled and used to
keep track of abstract operations such as data migration,
h) conditional expressions, grouped in several classes that represent
certain high-level constructions, are detected and reported,
i) providing run-time code modification capability to bypass
checks, anti-debugging code, simulate different run-time conditions,
etc,
j) providing an advanced step-by-step debugger that is fully integrated
with Fenris and provides modular capabilities - so if you want to
implement your own, very complex conditional breakpoint, all you have
to do is to load your module.
For recognized structures and calls, please refer chapter 0x05 (tracing
mechanism). Now, let's talk about front-end.
[0x02] How can I use it?
------------------------
Fenris is distributed under terms and conditions of the GNU public license. It
means you are free to use, distribute and modify this program. You can use it
in any way you want - to debug your own code, find bugs in third-party
software, understand and reverse-engineer protocols and algorithms, and such.
If you want to use this code in your own software, please let me know.
On compilation time, you can choose what variant of Fenris you want to build.
Some of more useful options:
make all - typical compilation; highly optimized, but build
process is time and memory consuming
make debug - debugging version; this enables memory tracing,
debugging symbols, disables all optimizations
and such; not recommended for everyday use.
Invoking main Fenris is pretty simple, and does not really require any
documentation for anyone who ever used 'strace' or similar application. Called
without any parameters, it will display short help. The main program is
invoked the following way:
fenris [ -E PAR=VAL ] [ -u user ] [ -o file ] [ -L dbase ] [ -R a:b ]
[ -t nnn ] [ -P ip:off:val ] [ -sdyiCSfFmGxpAeq ] program
[ params... ]
Mandatory parameter is program name, eventually followed by program
parameters. If, for some reason, program name has to start with '-', it should
be preceded with '--' parameter. Before program name, you can place one or
more optional parameters, such as:
-o filename
This options writes results to file instead of stderr. It is faster and
recommended in all cases.
-E PAR=VAL
Puts PAR in the environment. This is especially useful if you want to trace
a program with unusual LD_PRELOAD or other settings that would affect the
functionality of 'fenris' itself if modified earlier. Multiple -E options are
allowed.
-u user
Run as user. This option is available for root (see section 0x04, security
issues), and will cause program to effectively run with uids, gids and
supplementary groups of given user.
-R a:b
This option traces code from the moment when eip reaches point a to the moment
when it reaches b. Incomplete range can be provided - for example, -R
:0x12345678 will trace code from the beginning to eip 0x12345678, and -R
0x12345678: will start tracing at 0x12345678 and continue as long as possible.
NOTE: think of it as trigger points, not a continuous range. For example, if
you use -R 0x12345678:, but eip 0x12345678 is never reached, even if
0x23456789 is being executed, trace will never start.
This option is wonderful for starting trace at certain nest level and
continuing it until this execution level is exited.
-L dbase
Load additional (supplementary) fingerprints database. Multiple -L options
allowed. If filename does not contain slashes, Fenris will look for it in
directories described later in this section.
-t nnn
Main function is nnn rets from _do_global_ctors_aux. By default, this is set
to 2, and does not have to be changed unless something is really wrong. You
should use this option if you see that trace ends with '...return from main'
almost immediately at the beginning (try increasing -t parameter) or somewhere
in the middle or does not reach main at all (try decreasing). However, this
should not happen, in general. The only case I'm aware of are HMM 3 binaries
(patchlevel 1.3.1a, does not affect 1.3), they require -t 3 instead.
-X seg
Use this segment prefix instead of the default (determined for a typical
binary on your system) as a code segment. Code segment is the segment Fenris
actively traces. Some ELF binaries can be altered to start in a different
segment - a good example is a burneye ELF crypting tool. Code segment address
is used by Fenris for some operations, such as describing parameters, handling
signal handlers, function fingerprinting. While not absolutely necessary, it
is wise to pass this parameter when suitable. Pass the most significant byte
of code segment starting address as this parameter (for example, if your code
segment starts at 0x050a0000, use 0x05).
-P ip:off:val
This directive means: change a byte at address 'off' to 'val' when eip reaches
'ip'. If 'ip' is omitted or zero, this rule will be applied immediately to the
freshly mapped binary (keep in mind that some memory regions mapped later may
be not available at this moment). Read-only flag is generally overridden, and
for files mapped into memory in read-only mode, a local copy of the modified
page is spawned. All values passed to this parameter can be in decimal or in
hex if preceded with 0x, and multiple options are possible. Non-IP entries
will be applied only once, at the beginning. All others will be applied every
time a given IP is reached.
There are some additional considerations to be aware of when used in
conjunction with tracing across execve()s - see -e option description for
details.
-s
This option disables automatic prolog detection. It is not recommended, as it
makes ./fenris trace whole linking process and libc initialization. However,
in rare cases when binary is compiled on odd, not supported system, this might
be a solution. For long-term operations, however, it is recommended to contact
the author providing his with this binary (or parts of it), so he'll be able
to add support for this specific construction.
-y
Reports memory writes and reads immediately (without -y, memory access is
reported per function on return).
-C
Inhibits tracing conditional expressions. This option is useful if output will
be read by human, as it might decrease amount of reported information.
-S
Inhibits resolving library functions. This might effect in some speed
improvement, but is generally not recommended without a good reason.
-f
Trace child processes after fork() or vfork(). Might be useful for tracing
daemons and such (however it might cause some problems due to signal delivery
semantics changes, see 0x07, known bugs)
-d
Do not describe function parameters. Reduces amount of generated output.
-F
Do not fingerprint functions. This option is effective for static binaries
only, and will disable loading and displaying fingerprints. This is not really
recommended - for stripped binaries, it makes your life more difficult, for
binaries with symbols has almost no effect. However it might reduce memory
usage and improve speed.
-m
Do not trace memory writes. This option reduces amount of generated output.
-i
This option disables indenting, reporting of pid and nesting level. It makes
output non-structural, non-standard, but shorter. This will also break
compatibility with ragnarok.
-x
This option causes Fenris to ignore 'return from main' and to continue
tracing, returning to nest level 0. Generally speaking, this is not
recommended at any time. If you have problems with 'return from main'
appearing too early in the trace, try re-adjusting -t parameter instead. If
this do not help, apparently one or more of calling or return conventions used
by traced application are not supported, and you shouldn't rely on results
anyway.
-p
Prefix every message with eip. Some commands report eip, some not, this might
be useful for debugging, and is a must if you want to modify the code later
with -P option. This option is compatible with ragnarok. Note that information
is not displayed in some uniform way. For example, syscalls are displayed
after return, local functions are displayed before call - so it takes some
time to get the idea.
-A
Assume that all functions return some value, regardless of all other
conditions. This will trigger some meaningless return values reported, but is
useful if the binary is very optimized.
-q
Do not report last line of output to the debugger. This is meaningful only
with -W, and makes sense when you use a multi-window debugger shell that
already reports Fenris output (we're working on such a shell right now).
-G
"Go away" option. Can be used only in conjunction with -W, and it basically
turns of all analysis capabilities of Fenris - from tracing nesting level,
detecting function / library / system calls, thru many other capabilities. It
is useful for troublesome non-C code. Fenris output will be practically
completely disabled, and only some debugging messages will be supported (such
as single-step, getmem, address breakpoint, etc).
-e
Trace new code loaded by execve(). This option might be convenient in some
cases, but should be used with caution. Also, be warned that -P option will be
global and apply to both old and new image in memory, except for no-IP entries
that would be applied only once.
For more information on computer forensics applications, you may want to visit
http://lcamtuf.coredump.cx/fenris/reverse.txt, where I tried to give few hints
on approaching May 2002 reverse engineering challenge from Project Honeynet.
Managing fingerprints database is relatively simple. First of all, Fenris
looks for a database in the following places:
./fnprints.dat
$HOME/.fenris/fnprints.dat
$HOME/fnprints.dat
/etc/fenris/fnprints.dat
/usr/local/etc/fenris/fnprints.dat
Additionally, custom fingerprints database can be specified by -L option
(multiple databases allowed). Same search logic applies to -L parameters,
unless they contain path components ('/'). This is reasonable to maintain
separate fingerprint databases, as it allows you to be selective. For example,
if you are about to trace 'sash', you can be pretty sure it won't use libX*
libraries, so first, you can make lookups faster, and then, you minimize
eventual false positives or confusion caused by identifying some functions
incorrectly. As an example, I provide fingerprints for pretty old, but still
used glibc 2.0.7 in support/fn-2.0.7.dat, and fingerprints for libc5
(support/fn-libc5.dat). Note that, as for today, Fenris will probably not work
on libc5 systems (I have to port it), but this can be used against statically
linked binaries taken from such systems.
The main database shipped with Fenris right now is a composite database for
all major libraries for x86 libc 2.1.x and 2.2.x generated by gcc 2.9x to 3.1.
It is pretty huge, but also versatile. If you believe it makes sense to
maintain smaller libraries, feel free to do it and send me your selection!
Fingerprints database is a plain text file in the following format:
[debug info] function_name MD5_SIGN
Where 'debug info' is used by 'fprints' utility to indicate the source
(filename+offset) of given symbol, function_name is self-explanatory, and
MD5_SIGN is 8-digit hexadecimal MD5 shortcut for given function (see section
0x05, tracing mechanism for more details on hashing algorithm).
'fprints' utility accepts any ELF file (executable, shared library or
relocatable .o file / .a archive) as a parameter and generates signatures for
all functions. It does not really make any sense to grab signatures from
shared libraries, as they are not used to build static binaries, so you should
target .o files instead. However, it is possible and sometimes reasonable to
gather signatures from ELF executables. It allows you to fingerprint some
frequently used functions (e.g. __non_dynamic_init or some custom common code
used by others; let's say Loki uses some common engine for all their games,
you can easily index functions in this engine once and benefit from automated
recognition later). Typical output looks like that:
[printf.o+52] printf CC6E587C
[printf.o+52] _IO_printf CC6E587C
--> printf.o: done (2 functions)
As you see, one of entries is just an alias.
Selected 'fprints' results can be appended to fnprints.dat file of your
choice. It is important to mention that many libraries have multiple entries
for the same function, so 'fprints' shouldn't be really used to gather
fingerprints for large .a archives, like libc. This task can be accomplished
by invoking 'getfprints' utility, which is a shell script wrapper around
fprints. It can process whole .a archive or even multiple archives at once,
eliminate dupes, and such. Please note that it is perfectly possible to copy
.a files from a system that is not directly supported by Fenris, for example,
libc5 box, and extract signatures on a different system.
When invoked with no parameters, 'getfprints' will extract default set of
symbols from:
/usr/lib/libc.a
/usr/lib/libm.a
/usr/lib/libdl.a
/usr/lib/libresolv.a
/usr/lib/libreadline.a
/usr/lib/libtermcap.a
/usr/lib/libssl.a
/usr/lib/libBrokenLocale.a
/usr/lib/libcrypt.a
(one static binary)
This is the way it is invoked by the Makefile script, and can be used at any
time to restore defaults or to update signatures (for new libc version, for
example). If invoked with one parameter, 'getfprints' will go thru this .a
file or set of .a files. An example would be:
./getfprints "/usr/lib/libcrypto.a /usr/lib/libmd5.a"
It is important to quote the list so it effectively makes one parameter.
Otherwise, only first file will be processed. Call it laziness on my end ;-)
Default output file for 'getfprints' is fnprints.new in current directory.
When integrating it with existing fnprints.dat, please make sure you eliminate
dupes by issuing the following command:
cat fnprints.new fnprints.dat | sort | uniq >clean-new.dat
This utility requires ./fprints to be in current directory or in your path.
Another tool provided with the project is called 'dress', roughly an opposite
to 'strip'. It will accept a stripped static ELF binary as a parameter, and
will try to detect library functions. Detected names will be placed in the
symbol table and a new ELF file will be generated. Usage is rather simple:
./dress input_elf - this will dump symbols to stdout
./dress input_elf output_elf - this will create a new ELF with symbols
Additional options:
-f nnn - use this file for fingerprint database
-s xxx - use this name as a code section (override .text)
Note that symbols generated are not GDB debugging info. In other words, you
can view them with nm, objdump, they will be shown in gdb disassembly, but you
might have problems setting an explicit breakpoint such as "break printf".
Blame GDB. As a workaround, you can run dress without a second parameter once
again, and grab interesting addresses from the output. Enjoy.
Note that 'dress' has nothing to do with 'unstrip', which is used to, quote,
"replace the symbol table in dynamically linked executables".
The last component discussed here is Aegir, the interactive debugger. For
information for programmers, please refer to doc/debug-api.txt. This brief
write-up should help you with developing modules for Aegir or even replacing
it with your own debugging shell in easy way.
Whole interactive debugging functionality in Fenris is designed to provide
instruction-by-instruction, breakpoint-to-breakpoint and
watchpoint-to-watchpoint capabilities within the local code. This means that
while it is possible to set up a breakpoint in library code, it is not really
possible to walk thru library functions instruction by instruction. This is
done for your own good, I doubt you really want to debug libc with Fenris.
Fenris does not trace nesting level, function calls and so on within libc, so
your possibilities are very limited anyway. Keep in mind that Fenris is an
executable tracker, not a library debugger, and will treat library space
pretty much like kernel space - a black hole. We are not trying to understand
library functions, they are documented and predictable (note: this, obviously,
won't be true for suspected code loaded as a shared library; Fenris will
support such code in the future).
Right now, I'm going to focus on front-end functionality. Running Aegir is
very straightforward, as all parameters are controlled by whatever you passed
to Fenris, and the only parameter you have to pass is the path you gave to
Fenris using -W option earlier. Fenris MUST be already running with -W option
to launch Aegir. Aegir will shut down as soon as Fenris exits. Aegir provides
some basic gdb-alike functionality, but also several more interesting
features. In its current version, it also lacks several features, such as
support for symbolic names in many functions, which can be a minor annoyance
and should be fixed in 0.07.
The GUI version of Aegir, nc-aegir, works basically the same way, but provides
an organized debugging screen with register, memory and code views, integrated
Fenris output view, and automatic control over Fenris parameters. nc-aegir
integrates Fenris session with the debugger, and it uses either 'screen'
utility (when running on a text terminal) or xterm session (when running under
X Window system) to provide comfortable multi-view debugging environment. The
GUI is documented by a self-explanatory help available after pressing Alt-H,
so I will not cover it extensively here.
Please note that there is a fundamental difference between how Aegir
/ nc-aegir and gdb handle interruptions. If you hit Ctrl+C in Aegir or
nc-aegir, or issue a "stop" command, it will not stop immediately if the
process is in the middle of a blocking call. It will schedule stop for as soon
as the control returns to userspace. This is to avoid problems with
interrupted syscalls, gdb style. To terminate the program immediately, hit
Ctrl+C again, or use "halt" command. You will be then instructed whether the
syscall will resume upon continuation or not, and what to do to avoid
problems.
Once Aegir is running, you should have access to its internal help, and all
messages are rather user-friendly. The following list of commands is provided
for more detailed reference:
- dynamic
This is probably the first command to issue for a standard dynamically linked
binary. Fenris stops at first instruction, which, for dynamic executables,
would be the linker. To skip whole linking process and libc prolog, simply
type "dyn" and wait a while. Of course, nothing stops you from walking thru
the linking process and libc entry, but in most standard applications, it is
pointless. On the other hand, it can happen that ELF is tweaked to hide some
code in this phase, before "main" is reached, so this feature is not automatic
in interactive debugging mode.
On some system, execution will stop at the very end of libc intro, and
additional "ret" might be necessary.
- disass [ x [ len ] ]
Called without parameters, will provide a disassembly of the instruction at
current eip. Called with one parameter, will disassemble one instruction at
any given address. With two parameters, will disassemble "len" bytes starting
at address x. Disassembly uses something that should match AT&T assembler
notation, and all direct addresses are associated with their symbolic names,
if any found. Note that "disass", like most other commands, does not
understand symbolic names passed instead of 'x'. In other words, you can't
just type "disass function_foo", this is a limitation of current Aegir
implementation. On the other hand, you can use "info" directive to look up an
address for a given name.
Note that this command called with a single parameter in nc-aegir will change
the view in code window instead of disassembling to the console.
- regs
Shows general purpose registers. Note that Fenris does not really support
floating point commands in any way (thanks to its internal disassembler), and
I decided not to include fp registers in Aegir for now. Most of registers are
displayed in hex and decimal; eflags are displayed as hex and octal.
- back
Displays stack backtrace - calls history. Note that what happens in libc is
not covered here. If you set up a breakpoint on syscall "nanosleep", and this
syscall is called from a library function called from another library
function, all you'll see in stack backtrace will lead to the point where first
library function was called. Backtrace includes stack address range that
belongs to this function and other information, such as from where was it
called. This capability is not affected by -fomit-frame-pointer, or any other
options that can confuse gdb.
- cur
Displays last output from Fenris. "Last output" stands for last "output
entity", which typically means last sequence of output caused by some code
construction. One instruction, such as RET, can result in multiple lines being
written by Fenris. All of them are considered a single entity. But if next
instruction generates another line, this line is considered a new entity.
Fenris generally reports back to Aegir the last entity produced before
a running process reached a breakpoint. "Cur" will return this entity until
the process is continued and stopped once again, and any message from Fenris
was generated in between. This mechanism is a bit complex, but works pretty
well. You probably don't want to get all lines from Fenris on your debugging
console, but perhaps would appreciate knowing where you stopped.
- info x
Displays the information associated with name or address x. First, if x is
non-numeric, the address associated with 'x' is resolved, then, additional
information about this address is obtained. This additional information is
what Fenris knows about the address - associated name, first sight, last
modification, size.
- fdinfo x
Displays what Fenris knows about file descriptor x. This typically includes
associated file / socket, and first sight record.
- break x
Sets a breakpoint at address x.
Note: breakpoints are ignored inside libc, except for ones being set at the
beginning of a libcall. This way, you can breakpoint at 'printf', but it is